[EAAPL-RAG007] Agentic Retrieval-Augmented Generation

Category: Artificial Intelligence / Retrieval-Augmented Generation Sub-category: Agentic and Multi-Hop RAG Version: 1.2 Maturity: Proven Tags: rag agentic multi-hop iterative-retrieval query-planning self-critique tool-use reasoning Regulatory Relevance: EU AI Act Article 14 (Human oversight for autonomous systems), ISO/IEC 42001 Section 8.5 (AI system operation), NIST AI RMF (Govern 1.7 — autonomous decision-making)

1. Executive Summary

Agentic RAG places a reasoning AI agent in the orchestration loop of the retrieval-generation pipeline, enabling it to plan multi-step retrieval strategies, evaluate the quality of retrieved evidence, execute iterative searches to fill gaps, and self-critique its nascent answers before returning a final response. Unlike standard RAG, which executes a single fixed retrieval cycle per query, Agentic RAG can break a complex question into sub-questions, retrieve evidence for each, synthesise intermediate answers, identify what it still doesn't know, and retrieve again — autonomously, until it has sufficient grounded evidence to answer with confidence.

For enterprise leaders, Agentic RAG addresses a category of knowledge queries that single-cycle RAG fundamentally cannot handle: multi-hop questions that require chaining across multiple documents ("Which of our suppliers are located in regions affected by the sanctions announced in the regulatory alert published this week, and what are our contractual obligations for those suppliers?"), questions requiring synthesis across multiple evidence sources, and research-style queries that require iterative exploration of a knowledge domain. The business value is the automation of knowledge work that previously required a skilled analyst to retrieve, read, synthesise, and reason across multiple documents — turning a 2-hour research task into a 30-second AI-assisted workflow.

2. Problem Statement

Business Problem

Single-cycle RAG systems retrieve a fixed set of candidate documents and generate an answer from them in one pass. This architecture is adequate for factual lookup questions but insufficient for analytical questions that require multi-document synthesis, causal reasoning, or iterative hypothesis refinement. Analysts who use RAG systems for complex research tasks frequently report that the system misses relevant documents that would only be found if the initial answer had been used to formulate a better search query.

Technical Problem

Standard RAG has no mechanism for the system to recognise that its retrieved context is insufficient to answer the question, no ability to plan a multi-step retrieval strategy before retrieving, and no capacity to iteratively refine its retrieval based on partial answers. The single retrieval cycle architecture is fundamentally limited for complex, multi-hop knowledge questions.

Symptoms

• RAG system answers complex analytical questions with shallow, incomplete responses that reference only one or two source documents
• Users manually perform "follow-up searches" after receiving an initial RAG answer, indicating the system did not exhaust the relevant knowledge
• High analyst override rate: AI-generated answers are frequently edited with additional context that the analyst had to find separately
• The system answers "I don't have enough information" for questions that are answerable from the corpus but require multi-hop reasoning

Cost of Inaction

• Complex knowledge work remains manual and unautomated, despite a capable knowledge corpus
• Analysts treat the RAG system as a first-pass tool only, not as a research assistant capable of deep synthesis
• Competitive disadvantage versus AI deployments that automate complex analytical workflows end-to-end

3. Context

When to Apply

• Complex multi-hop questions requiring evidence from multiple, sequentially discovered documents
• Research synthesis tasks: competitive analysis, regulatory impact assessment, risk analysis across multiple source documents
• Question answering where the answer to one sub-question determines which documents to retrieve next (dependent retrieval chains)
• Use cases where the appropriate retrieval strategy varies per question (some questions need one retrieval; others need five)
• Scenarios where the agent should be able to answer "I cannot find sufficient evidence" after exhausting retrieval strategies, rather than hallucinating

When NOT to Apply

• Simple factual lookups answerable in a single retrieval cycle (adds unnecessary latency and cost)
• Latency-critical applications (agentic loops add 1–10 seconds of reasoning time per iteration)
• Autonomous action-taking scenarios where the agent's retrieval findings would directly trigger writes or external API calls without human approval — this requires explicit Human-in-the-Loop governance (EAAPL-RAG007 combined with HITL patterns)
• Corpora that are too small to benefit from multi-hop retrieval (< 10,000 documents)

Prerequisites

• An underlying RAG retrieval capability (EAAPL-RAG001 or EAAPL-RAG005)
• An LLM with reliable tool-calling / function-calling capability (GPT-4o, Claude 3.5, Gemini 1.5 Pro)
• A defined set of retrieval tools the agent can invoke (search, filter, summarise, compare)
• A maximum iteration limit to prevent infinite loops
• Observability instrumentation that traces each agent decision, tool call, and its output

Industry Applicability

4. Architecture Overview

Agentic RAG wraps the retrieval-generation pipeline in a Reason-Act-Observe loop (a specialisation of the ReAct framework). The agent is an LLM with access to a defined set of retrieval tools and a scratchpad for recording its reasoning steps.

Query Planning

When a complex question is received, the agent's first step is query planning: it reasons about what information is needed to answer the question and decomposes the original query into a set of sub-queries with dependencies. For example, "Which of our suppliers are in sanctioned regions per this week's alert?" decomposes into: (1) retrieve this week's sanctions alert, (2) extract the list of sanctioned regions, (3) retrieve the supplier list, (4) cross-reference supplier regions against the sanctioned list. The query plan is recorded in the agent scratchpad and drives the retrieval sequence.

Query planning can be explicit (the agent writes a multi-step plan before executing any retrieval) or implicit (the agent uses tool calling in sequence, with each tool output informing the next call). Explicit planning produces more predictable and auditable behaviour; implicit sequential tool calling is more flexible but harder to trace.

Retrieval Tools

The agent is given a defined set of retrieval tools as callable functions:

• search(query: str, filters: dict) → List[Chunk]: standard RAG retrieval
• get_document(doc_id: str) → Document: retrieve a full document by ID (for follow-up reading)
• summarise_results(chunks: List[Chunk]) → str: summarise a set of retrieved chunks
• compare_documents(doc_ids: List[str], aspect: str) → str: compare specific aspects across multiple documents
• extract_entities(chunks: List[Chunk], entity_type: str) → List[str]: extract named entities for use as next-query parameters

Tools are strictly typed and validated — the agent cannot invoke arbitrary code, only the defined tool set. Tool definitions include descriptions that guide the agent's tool selection.

Iterative Retrieval and Self-Critique

After each retrieval cycle, the agent evaluates its current evidence against the original question using a self-critique prompt: "Given the question and the evidence retrieved so far, what information is still missing? What additional searches would improve the answer?" If the self-critique identifies gaps, the agent formulates and executes additional retrieval calls. The loop continues until one of three stopping conditions is met: (1) the agent's self-critique determines the evidence is sufficient, (2) the maximum iteration limit is reached, or (3) additional retrieval is producing diminishing returns (identical or near-identical results to previous retrieval steps).

Grounded Final Synthesis

When the retrieval loop concludes, the agent synthesises a final answer from the accumulated evidence. The synthesis step is more complex than in standard RAG: the agent must integrate evidence from multiple retrieval rounds, handle potentially conflicting evidence, attribute each claim to its source, and structure the answer appropriately (summary, list, table, or prose depending on the question type). The synthesis prompt explicitly instructs the agent to cite each claim and to acknowledge when evidence is incomplete or conflicting.

Human Oversight Gate

For high-stakes agentic tasks (those whose answers will directly inform consequential decisions), an optional human oversight gate is inserted before final response delivery. The gate presents the agent's reasoning trace, the evidence sources used, and the draft answer to a human reviewer, who can approve, edit, or reject the answer. This gate is a regulatory requirement for high-risk AI use cases under EU AI Act Article 14.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Flow

Error Flow

8. Security Considerations

Tool Boundary Enforcement

The most critical security control in Agentic RAG is ensuring the agent cannot invoke tools outside its defined tool set. All retrieval tools are read-only — the agent must have no access to write, update, or delete operations. Tool definitions must be validated against a schema; any tool invocation not matching a defined schema must be rejected. The agent runtime must run in a sandboxed environment with no outbound network access beyond the defined tool API endpoints.

Prompt Injection in Multi-Hop Context

Multi-hop retrieval creates a compounded prompt injection risk: an adversarial document retrieved in hop 1 could inject instructions into the agent's scratchpad that influence hop 2 onwards. The agent system prompt must explicitly instruct the model that retrieved content is data, not instructions, and the orchestrator must sanitise tool outputs before inserting them into the agent context window.

OWASP LLM Top 10 Mitigations

9. Governance Considerations

Autonomous Decision Boundary

Agentic RAG must have a clearly defined boundary between autonomous retrieval (permitted) and autonomous action-taking (not permitted in this pattern). The agent may search, read, summarise, and synthesise — but the final answer must be delivered to a human, not used to trigger downstream automated actions without explicit human approval.

Reasoning Trace as Governance Artefact

Every agentic session must produce a complete, immutable reasoning trace: the initial query, each reasoning step, each tool call with its parameters, each tool output, and the final synthesis. This trace is the primary governance artefact for post-hoc review of agentic decisions. Regulators reviewing an AI-assisted compliance analysis must be able to trace every claim back to the retrieved document that supported it.

Governance Artefacts

10. Operational Considerations

Monitoring

Service Level Objectives

11. Cost Considerations

Cost Drivers

Indicative Cost Range

12. Trade-Off Analysis

Agentic Orchestration Framework Comparison

Query Planning Strategy

Architectural Tensions

13. Failure Modes

14. Regulatory Considerations

15. Reference Implementations

AWS

• Agent: Bedrock Agents (native multi-hop retrieval) or LangChain on Lambda
• Retrieval tool: Amazon Kendra or OpenSearch k-NN
• Scratchpad: Amazon ElastiCache (Redis) for session state
• Audit: CloudWatch Logs with structured JSON; X-Ray for trace

Azure

• Agent: Azure AI Studio Prompt Flow with agent orchestration, or LangChain on Azure Functions
• Retrieval tool: Azure AI Search
• Scratchpad: Azure Cache for Redis
• Audit: Azure Monitor + Application Insights

GCP

• Agent: Vertex AI Agent Builder (Grounding with Search) or LlamaIndex on Cloud Run
• Retrieval tool: Vertex AI Vector Search
• Scratchpad: Cloud Memorystore (Redis)
• Audit: Cloud Trace + Cloud Logging

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Proven — Agentic RAG is deployed in production for research and compliance use cases; ReAct and Plan-and-Execute frameworks are mature; the primary ongoing challenges are iteration cost management and reasoning trace audit quality.

18. Revision History