[EAAPL-RAG006] Streaming Retrieval-Augmented Generation

Category: Artificial Intelligence / Retrieval-Augmented Generation Sub-category: Real-Time and Event-Driven RAG Version: 1.1 Maturity: Proven Tags: rag streaming real-time event-driven freshness kafka time-windowed-retrieval change-data-capture Regulatory Relevance: APRA CPS230 (operational resilience for real-time systems), MiFID II (real-time data obligations in financial services), ISO/IEC 42001 Section 8.4

1. Executive Summary

Streaming RAG extends the foundational RAG architecture to continuously ingest and index data from real-time event streams, enabling language model responses to be grounded in current, up-to-the-minute knowledge. While standard RAG operates on a periodically refreshed static corpus, Streaming RAG treats the knowledge index as a continuously evolving data product where events (new documents, document updates, market data changes, operational alerts) are reflected in retrievable context within seconds to minutes of occurrence.

For business leaders in time-sensitive domains — financial trading, real-time customer operations, industrial IoT, news and media — the distinction between a RAG system that knows what happened yesterday versus one that knows what happened five minutes ago can determine whether the AI assistant is fit for operational use. A customer service AI grounded only in yesterday's product data will give incorrect answers about today's promotional pricing. A risk management AI that cannot access this morning's regulatory alerts may miss a material compliance event. Streaming RAG closes this gap by making knowledge freshness a first-class architectural concern, traded off explicitly against retrieval quality and operational complexity.

2. Problem Statement

Business Problem

Standard batch-ingested RAG systems have knowledge that is hours or days stale. For many enterprise use cases this is acceptable; for many others it is not. Customer service assistants need to know today's pricing, promotions, and service status. Operations assistants need to know the current status of incidents, deployments, and SLA breaches. Financial services assistants need to know this morning's rate changes and regulatory updates. A RAG system that cannot reflect these changes in real-time is operationally unusable for these use cases.

Technical Problem

The batch ingestion architecture of standard RAG introduces inherent latency between when a document is created or updated and when it becomes retrievable. Document crawling, format parsing, chunking, embedding, and index upsert are sequential pipeline steps that typically take minutes to hours per document in a batch schedule. Event-driven and streaming systems demand a different architecture: an ingestion pipeline triggered by events rather than schedules, optimised for low-latency single-document processing rather than high-throughput batch processing.

Additionally, real-time data creates a freshness/quality tension: a document ingested seconds ago has not undergone the quality validation, editorial review, or metadata enrichment that a batch-ingested document has. The retrieval strategy must account for this — freshness and quality are both relevant signals for ranking, and the appropriate balance depends on the use case.

Symptoms

• AI assistant gives incorrect answers because its knowledge is stale (e.g., cites a policy that was updated this morning)
• Customer service AI provides yesterday's pricing to customers who have received today's promotional offer
• Operations runbook AI cannot answer questions about the current production incident because the incident report was just created
• News and media AI continues to reference an old version of a story after a material retraction was published

Cost of Inaction

• Customer complaints and churn from incorrect information delivery in real-time service channels
• Regulatory exposure: financial services AI providing stale rate or compliance data
• Operational risk: incident response AI grounded in stale runbooks during a live incident
• Trust erosion: users who notice knowledge staleness lose confidence in the AI assistant and revert to manual lookup

3. Context

When to Apply

• AI assistants for customer-facing channels where product, pricing, or service data changes frequently (daily or faster)
• Operations and incident response assistants where runbook and incident data must be current
• Financial services AI where market data, rate sheets, and regulatory alerts must be reflected within minutes
• News, media, and content platforms where articles, corrections, and editorial updates are continuous
• IoT and industrial operations where equipment status, alerts, and sensor thresholds change continuously

When NOT to Apply

• Knowledge corpora that change less frequently than once per day (standard batch ingestion is simpler and less expensive)
• Use cases where answer freshness is not a user requirement (historical research, legal document review)
• Environments lacking a mature event streaming infrastructure (Kafka, Kinesis) — the operational overhead of deploying streaming infrastructure solely for RAG is rarely justified
• Use cases where raw streaming data quality is insufficient for grounded AI generation (malformed events, incomplete records)

Prerequisites

• An event streaming platform (Apache Kafka, Amazon Kinesis, Azure Event Hubs, Google Pub/Sub) producing document change events
• Change Data Capture (CDC) tooling for database-originated changes (Debezium, AWS DMS)
• A vector database that supports low-latency upsert operations (Weaviate, Qdrant, Pinecone — all support real-time upsert)
• An embedding inference service with low-latency API (< 100ms per chunk) to avoid ingestion lag
• Freshness metadata (created_at, updated_at timestamps) that must be indexed alongside vectors

Industry Applicability

4. Architecture Overview

Streaming RAG introduces an event-driven ingestion pipeline that runs continuously alongside (or instead of) the batch ingestion pipeline. The two pipelines share the same vector database but differ in their triggering, throughput optimisation, and quality guarantees.

Event-Driven Ingestion Pipeline

The event-driven pipeline is triggered by document change events rather than a schedule. Events are consumed from a streaming platform (Kafka topic, Kinesis stream, or event hub). Each event carries the document content (or a reference to it) and a change type (CREATE, UPDATE, DELETE). The pipeline processes events through the standard chunking → embedding → upsert sequence but is optimised for low per-event latency rather than high batch throughput.

Key design decisions for the event-driven pipeline:

• Exactly-once semantics: document CREATE and UPDATE events must be processed exactly once to avoid duplicate vectors. Use idempotent upserts (upsert by document ID, not insert) and transactional offset commits in Kafka.
• DELETE propagation: when a document is deleted, all its chunks must be deleted from the vector index. This requires a chunk-to-document mapping in a metadata store (the document store serves this purpose).
• Partial document updates: if only a section of a document changes, a partial update strategy re-chunks and re-embeds only the changed sections, significantly reducing embedding compute cost for large documents.
• Event ordering: out-of-order events (UPDATE before CREATE due to partition skew) must be handled. Idempotent upserts naturally handle this; version numbers in event payloads allow stale-write detection.

Freshness-Aware Retrieval

The retrieval layer must incorporate document freshness as a ranking signal alongside semantic relevance. Two approaches are common:

1. Hard time-window filtering: pre-retrieval metadata filter restricts results to documents updated within a configurable time window (e.g., last 24 hours for market data; last 7 days for policies). This is simple and guarantees freshness but may exclude relevant older documents.
2. Freshness-decay scoring: the retrieval score is multiplied by a time-decay function exp(-λ × age_in_days) where λ controls the decay rate. Freshness is a soft preference, not a hard cut-off. Documents updated in the last hour score near-full relevance; documents updated a month ago have discounted relevance.

The appropriate strategy depends on the use case: real-time market data use cases benefit from hard time-window filtering; policy and procedure use cases (where a 3-year-old policy is still valid) benefit from freshness-decay scoring.

Index Consistency During Real-Time Upserts

Real-time vector upserts create a brief window of index inconsistency while the HNSW index is being updated. Most enterprise vector databases handle concurrent reads and writes gracefully (MVCC-style), but very high-throughput streaming scenarios may require index partitioning strategies (e.g., separate "hot" index for recent documents, "cold" index for stable historical documents) with a merge retrieval strategy. The hot/cold partition approach also enables different HNSW tuning parameters: the hot index is tuned for high write throughput; the cold index is tuned for high recall.

Quality Considerations for Streaming Data

Raw event data is often lower quality than batch-processed documents: events may be incomplete, may contain duplicate or near-duplicate content, or may represent intermediate states (a draft document, an unreviewed news post). A lightweight quality filter at the event consumer should enforce minimum quality thresholds (minimum document length, no-duplicate check via content hash, optional classification check) before triggering embedding and indexing.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Flow

Error Flow

8. Security Considerations

Real-Time ACL Update Propagation

In streaming RAG, documents can be ingested faster than ACL metadata is normalised. The stream consumer must reject document events that lack ACL metadata rather than ingesting with empty ACLs (which would make the document accessible to everyone). A document with no ACL metadata is ingested into a "pending ACL" queue and indexed only once ACL metadata is resolved.

Event Stream Security

The event streaming platform must enforce topic-level access control (Kafka ACLs or IAM policies for managed services). The embedding service API key must be stored in a secrets manager and rotated regularly. Event payloads containing highly sensitive content should be encrypted in transit with envelope encryption.

OWASP LLM Top 10 Mitigations

9. Governance Considerations

Freshness SLA Governance

The ingestion lag SLA (time from event publication to document retrievability) must be formally agreed and monitored. For each source type, the SLA should be documented in the Source Inventory (EAAPL-KNW003). Breach of the freshness SLA for Tier 1 sources must trigger an automated alert.

Governance Artefacts

10. Operational Considerations

Monitoring

Service Level Objectives

Disaster Recovery

11. Cost Considerations

Cost Drivers

Indicative Cost Range

12. Trade-Off Analysis

Freshness vs. Retrieval Quality

Event Granularity for Streaming

Architectural Tensions

13. Failure Modes

14. Regulatory Considerations

15. Reference Implementations

AWS

• CDC: AWS DMS → Amazon Kinesis Data Streams
• Streaming consumer: AWS Lambda (Kinesis trigger) + Amazon Bedrock Titan Embeddings
• Hot index: Amazon OpenSearch Service (real-time upsert)
• Cold index: Amazon Aurora pgvector
• Freshness decay: Lambda post-processor

Azure

• CDC: Azure Data Factory CDC → Azure Event Hubs
• Streaming consumer: Azure Functions (Event Hub trigger) + Azure OpenAI embeddings
• Hot/cold index: Azure AI Search (hot namespace + cold namespace)
• Merge layer: Azure Function combining hot/cold results

GCP

• CDC: Datastream → Google Pub/Sub
• Streaming consumer: Cloud Run (Pub/Sub push) + Vertex AI embeddings
• Hot index: Vertex AI Vector Search (real-time upsert supported)
• Cold index: AlloyDB pgvector

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Proven — Event-driven RAG ingestion is deployed in production at financial services and e-commerce scale; freshness-aware retrieval patterns are well-understood; hot/cold index partitioning is operationally mature.

18. Revision History