[EAAPL-RAG008] Multimodal Retrieval-Augmented Generation

Category: Artificial Intelligence / Retrieval-Augmented Generation Sub-category: Multimodal and Cross-Modal Retrieval Version: 1.0 Maturity: Emerging Tags: rag multimodal vision image-retrieval table-understanding cross-modal clip colpali document-intelligence Regulatory Relevance: EU AI Act Article 10 (Data quality across modalities), ISO/IEC 42001 Section 6.1, NIST AI RMF (Map 1.5)

1. Executive Summary

Multimodal RAG extends the retrieval-augmented generation paradigm to knowledge corpora that include images, charts, diagrams, tables, and other non-text modalities alongside prose documents. Enterprise knowledge frequently exists in forms that standard text-based RAG cannot access: engineering diagrams, medical imaging reports, financial charts, product photographs, scanned contracts, and slide decks with data visualisations. Multimodal RAG enables users to query this knowledge in natural language and receive answers grounded in visual as well as textual evidence.

For enterprise leaders in engineering, healthcare, manufacturing, finance, and legal domains, Multimodal RAG unlocks a significant portion of the knowledge corpus that text-only RAG leaves inaccessible. A maintenance engineer asking "What does the valve assembly look like on model X?" needs a diagram, not a text description. A financial analyst asking "Show me the revenue trend chart from the Q3 investor presentation" needs the chart itself, retrieved and analysed, not a text summary of revenue figures. The pattern is emerging rather than mature — the enabling technologies (multimodal embedding models, vision-language models capable of grounded QA over retrieved images) are advancing rapidly but have not yet reached the operational reliability of text-only RAG. Early adopters with visual-heavy corpora should pilot carefully and plan for ongoing model upgrades.

2. Problem Statement

Business Problem

Enterprise document corpora are not exclusively textual. Technical manuals contain engineering diagrams. Financial reports contain charts. Contracts contain tables of fees and conditions. Training materials contain screenshots. Product documentation contains photographs. Text-only RAG silently ignores all of this visual content, leaving entire knowledge domains inaccessible to AI-assisted search and Q&A.

Technical Problem

Text embedding models cannot embed images. Vector similarity search over text embeddings cannot retrieve images by semantic query. Vision-language models capable of answering questions from image content exist but require grounded evidence retrieval — they cannot answer questions about a diagram without the diagram being present in context. The architecture must therefore solve two distinct problems: (1) how to retrieve the most relevant image/diagram/table for a given query, and (2) how to present it to the generation model in a form that enables grounded visual question answering.

Symptoms

• AI assistant cannot answer questions about diagrams, charts, or photographs even though they exist in the knowledge corpus
• Users receive text-only answers to questions that require visual evidence, and must manually locate the relevant diagram
• RAG system quality evaluations show low recall on questions derived from figure captions, table contents, or diagram annotations
• Users explicitly request "show me the diagram" or "retrieve the chart" and the system cannot comply

Cost of Inaction

• Significant portions of the knowledge corpus remain unsearchable via AI, limiting the ROI of the RAG investment
• Engineers, clinicians, and financial analysts must perform manual visual search in parallel with AI text search, duplicating effort
• Competitive disadvantage as multimodal AI capabilities become standard in enterprise knowledge platforms

3. Context

When to Apply

• Knowledge corpora where more than 10% of content value is in non-text form (images, diagrams, charts, tables)
• Technical documentation with engineering diagrams, schematics, or photographs
• Financial document Q&A where charts and tables are primary information carriers
• Healthcare document Q&A over radiology reports, clinical diagrams, or pharmaceutical product photographs
• Legal contracts with tabular terms, fee schedules, and signature pages

When NOT to Apply

• Text-only corpora where all visual content is purely decorative (logos, page backgrounds)
• Deployments with strict latency requirements where multimodal embedding and vision-language model inference adds unacceptable overhead
• Organisations without a mature multimodal data ingestion and storage capability — text-only RAG should be deployed first

Prerequisites

• A multimodal embedding model capable of embedding both text queries and image content in the same vector space (CLIP, ColPali, Nomic Embed Vision)
• A vision-language model capable of grounded visual question answering (GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet — all support image input)
• An image/figure extraction pipeline (extracts figures and tables from PDFs and documents)
• Object storage for image assets (referenced from the vector index by URL or object key)
• A table understanding component (converts tables to structured JSON or markdown for LLM consumption)

Industry Applicability

4. Architecture Overview

Multimodal RAG introduces two new ingestion paths and one new retrieval path alongside the standard text pipeline. Understanding the distinct characteristics of each modality is essential to designing an effective architecture.

Multimodal Document Parsing

The ingestion pipeline must first extract non-text elements from documents. For PDFs, this requires a document parsing step that identifies figures, tables, and images and extracts them as separate assets. Tools such as Apache Tika, AWS Textract, Azure Document Intelligence, or Google Document AI can extract figures and tables from PDFs with varying quality. Each extracted asset is assigned a unique asset ID, a reference to its parent document, a page number, a caption (if present), and surrounding context text (the paragraphs immediately before and after the figure in the source document).

Image Embedding Path

Each extracted image is embedded using a multimodal embedding model that produces vectors in the same semantic space as text embeddings. CLIP (Contrastive Language-Image Pre-training) is the canonical architecture: trained on image-text pairs, it produces comparable embeddings for text queries and images, enabling cross-modal retrieval ("text query → retrieve relevant images"). ColPali is an emerging alternative that produces multi-vector patch-level embeddings for higher-resolution document understanding.

Images are stored in object storage (S3, Azure Blob, GCS) and referenced by URL in the vector index. The vector index entry contains the image URL, the image's embedding, the parent document ID, the caption, and surrounding context text. Storing the full image bytes in the vector database is not recommended — only references.

Table Understanding Path

Tables require a distinct handling path because they are structured data, not natural language. Table extraction (via document intelligence services) produces structured table representations. These are then converted to either Markdown table format (for LLM consumption) or JSON (for structured query interfaces). The table as a whole is embedded as a text embedding (of its Markdown representation) for retrieval, not as an image.

Cross-Modal Retrieval

At query time, the user's text query is embedded using the multimodal embedding model and used to search both the text chunk index and the image/table index simultaneously. Scores from both indexes are merged (using RRF or weighted combination) and the top-K results across all modalities are selected. The context assembler then constructs a multimodal prompt that includes both text chunks and images (base64-encoded or URL-referenced, depending on the VLM API).

Vision-Language Model for Grounded Visual QA

The generation step uses a vision-language model (VLM) that accepts both text and images in its context window. The VLM is instructed to answer the user's question based on the provided text and visual evidence, citing both text sources and image sources. The prompt structure places the visual evidence (images, tables) alongside the text chunks and explicitly asks the model to ground its response in the visual content when relevant.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Flow

Error Flow

8. Security Considerations

Image Content Classification

Images in enterprise documents may contain sensitive content (facial photographs, handwritten signatures, confidential diagrams). Image classification must be applied at ingestion to flag sensitive images and enforce the same ACL-based access controls as text documents (EAAPL-RAG003). A document with a PROTECTED classification propagates that classification to all extracted images.

OWASP LLM Top 10 Mitigations

9. Governance Considerations

Visual Content Governance

All extracted images must be inventoried as part of the corpus inventory (EAAPL-KNW003). Images containing personal information (photographs, handwritten documents) require explicit privacy assessment. The image extraction pipeline must be reviewed by the Privacy Officer before processing HR or healthcare documents.

Governance Artefacts

10. Operational Considerations

Monitoring

Service Level Objectives

11. Cost Considerations

Cost Drivers

Indicative Cost Range

12. Trade-Off Analysis

Multimodal Embedding Approach

Table Handling Strategy

Architectural Tensions

13. Failure Modes

14. Regulatory Considerations

15. Reference Implementations

AWS

• Document Intelligence: Amazon Textract (figure + table extraction)
• Image storage: Amazon S3
• Multimodal embedding: Amazon Titan Multimodal Embeddings G1 or self-hosted CLIP on SageMaker
• Image vector index: Amazon OpenSearch Service with k-NN
• VLM: Amazon Bedrock (Claude 3.5 Sonnet or Nova)

Azure

• Document Intelligence: Azure AI Document Intelligence (figure + table extraction)
• Image storage: Azure Blob Storage
• Multimodal embedding: Azure OpenAI (CLIP via custom deployment) or Azure AI Vision
• Image vector index: Azure AI Search (vector mode)
• VLM: Azure OpenAI GPT-4o (native image input)

GCP

• Document Intelligence: Google Document AI
• Image storage: Google Cloud Storage
• Multimodal embedding: Vertex AI Multimodal Embeddings
• Image vector index: Vertex AI Vector Search
• VLM: Vertex AI Gemini 1.5 Pro (native multimodal)

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Emerging — Multimodal embedding models and VLMs with image input are production-grade (GPT-4o, Gemini 1.5 Pro); document intelligence for figure extraction is mature; end-to-end multimodal RAG pipelines are in early production at leading enterprises but tooling is less standardised than text-only RAG.

18. Revision History