[EAAPL-RAG004] Federated Retrieval-Augmented Generation

Category: Artificial Intelligence / Retrieval-Augmented Generation Sub-category: Distributed and Federated Retrieval Version: 1.1 Maturity: Emerging Tags: rag federated distributed data-sovereignty privacy-preserving cross-organisation score-normalisation Regulatory Relevance: GDPR Chapter V (Cross-border transfers), Privacy Act 1988 Part IIIA, EU AI Act Article 10, APRA CPS234 §55 (third-party data), Australian Data Sovereignty requirements

1. Executive Summary

Federated RAG enables retrieval across multiple organisationally or geographically distributed knowledge bases without requiring data to be centralised in a single repository. Each participating node maintains its own vector index and document store, enforces its own access controls, and responds to retrieval requests without exposing raw documents to the federation coordinator. A central orchestrator fans out queries to participating nodes, collects ranked result sets, normalises scores across heterogeneous indexes, and assembles the final context for the language model.

For CIOs navigating cross-border data regulations, multi-entity corporate structures, or public-private data sharing partnerships, Federated RAG provides a principled architecture for enabling AI-powered knowledge synthesis across organisational boundaries without violating data residency, sovereignty, or confidentiality obligations. The pattern is relevant to government agency networks (where each agency maintains its own data under separate legislative mandates), healthcare provider networks (where patient data cannot leave jurisdictions), joint ventures (where each party contributes knowledge without disclosing the full corpus to the other), and multi-cloud enterprise architectures (where data must remain within specific cloud regions due to contractual or regulatory constraints).

2. Problem Statement

Business Problem

Organisations increasingly need to answer questions that span multiple independent knowledge bases, but the data owners of those bases have legitimate legal, commercial, or regulatory reasons not to share raw content. A government department asked to answer a cross-agency compliance question should not need to copy another agency's classified data into its own systems. A hospital network should not centralise patient records from all member hospitals into a single repository to enable AI search. A joint venture should not require each partner to surrender commercial-in-confidence documents to a shared index.

Technical Problem

Standard centralised RAG requires all documents to be ingested into a single vector index. This is architecturally incompatible with data sovereignty requirements, cross-border transfer restrictions, organisational trust boundaries, and contractual non-disclosure obligations. Naive distributed solutions (running independent RAG systems and aggregating answers) compound the problem: scores from different indexes are not comparable, different indexes may use different embedding models, and there is no principled mechanism for cross-node relevance ranking.

Symptoms

• Stalled AI initiatives because legal and compliance teams cannot approve centralised data aggregation
• Separate, siloed AI assistants per business unit with no ability to answer cross-unit questions
• Manual "copy and paste from multiple systems" workflows for staff who need cross-domain answers
• Joint venture or consortium participants rejecting shared data infrastructure proposals

Cost of Inaction

• Inability to leverage AI for cross-organisational knowledge synthesis, leaving manual research workflows in place
• Missed regulatory intelligence: compliance teams unable to get cross-jurisdiction answers from distributed regulatory corpora
• Competitive disadvantage versus organisations that have successfully federated knowledge infrastructure

3. Context

When to Apply

• Cross-organisation AI knowledge sharing where raw data cannot be centralised (government agencies, healthcare networks, industry consortia)
• Multi-jurisdiction deployments where data must remain within specific geographic boundaries
• Joint ventures and public-private partnerships where each party retains data sovereignty
• Large enterprises with independent subsidiaries or business units that have separate data governance regimes
• Regulatory constraint: the organisation is subject to data localisation requirements (EU GDPR Chapter V, Australian data sovereignty) that prohibit centralised indexing

When NOT to Apply

• Data can be legally and contractually centralised (use EAAPL-RAG001 for significantly better retrieval quality)
• Single-tenant deployment with no cross-boundary requirements
• Latency requirements preclude network round-trips to remote nodes (federated retrieval adds 50–200ms per node)
• All nodes use incompatible embedding models and re-embedding is not feasible (score normalisation quality degrades severely)

Prerequisites

• Each participating node must expose a standardised retrieval API (REST or gRPC) that accepts a query vector and metadata filters and returns ranked result sets with scores and metadata
• Score normalisation requires nodes to use compatible (ideally identical) embedding models, or to implement score calibration
• A federation coordination layer (either a dedicated service or the primary node acting as coordinator)
• Inter-node network connectivity with appropriate security (mTLS, API keys, VPN tunnels between trusted nodes)
• Data sharing agreement or federation protocol specifying what metadata each node may expose

Industry Applicability

4. Architecture Overview

Federated RAG decomposes into three functional layers: local nodes (which own their data and serve retrieval requests), the federation coordinator (which orchestrates cross-node retrieval and normalises results), and the generation layer (which assembles the federated context and invokes the LLM).

Local Node Architecture

Each local node is a complete, independently operated RAG stack: document ingestion pipeline, chunking engine, embedding model, vector database, and a retrieval API endpoint. The node enforces its own access controls — the coordinator has no ability to bypass the node's ACL enforcement. The node exposes a retrieval endpoint that:

1. Accepts a query vector (already embedded by the coordinator) and optional metadata filters
2. Enforces local ACL for the requesting entity (the coordinator's identity, not the end user's)
3. Returns a ranked result set: [(chunk_text, metadata, score, chunk_id)] — crucially, never raw document bytes
4. Applies rate limiting per coordinator identity to prevent denial-of-service

The node's governance principle is: the node owner controls what the node reveals. The node may return only metadata and scores (no chunk text) for sensitive documents, requiring the coordinator to resolve context from an approved escrow, or to simply note "relevant classified content exists but cannot be included in context."

Federation Coordinator

The coordinator is the query orchestration layer. Upon receiving a user query, the coordinator:

1. Embeds the query using the shared embedding model
2. Determines which nodes to query based on the query's topic scope and the user's inter-node authorisation
3. Fans out the query vector in parallel to all relevant nodes (with a configurable timeout per node)
4. Collects ranked result sets from responding nodes
5. Applies score normalisation to make scores from different nodes comparable
6. Re-ranks the normalised result set to produce a unified top-K list
7. Assembles context from the returned chunk texts (subject to each node's disclosure level)
8. Records which nodes were queried, which responded, and which declined (for audit and transparency)

Score Normalisation

Score normalisation is the most technically complex component of Federated RAG. When nodes use the same embedding model, cosine similarity scores are on the same scale and can be compared directly. When models differ (which should be avoided but may be unavoidable), score normalisation is required: each node's score distribution is modelled (mean and standard deviation) using calibration queries, and scores are Z-normalised before cross-node ranking. Reciprocal Rank Fusion (RRF) provides an alternative that is model-agnostic and rank-based, requiring only a ranked list from each node rather than raw similarity scores.

Privacy-Preserving Retrieval Modes

For maximum privacy preservation, the coordinator can operate in a query vector obfuscation mode using techniques from privacy-preserving machine learning: the query vector is perturbed with calibrated noise before being sent to nodes (analogous to differential privacy), preventing nodes from reconstructing the original query. This mode trades some retrieval quality for stronger privacy guarantees — the node cannot infer the full semantic content of the query. This mode is recommended for inter-organisation federation where the coordinator does not fully trust all nodes.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Flow

Error Flow

8. Security Considerations

Inter-Node Trust Model

The federation coordinator authenticates to each node using mTLS mutual authentication. Each node maintains a whitelist of coordinator certificate fingerprints it trusts. This prevents unauthorised coordinators from querying nodes. Nodes do not trust the coordinator's claim about end-user identity — nodes enforce only the coordinator's identity against the inter-node access policy.

Data Minimisation in Federation Protocol

The retrieval API response should include the minimum data necessary: chunk text (if the node's data sharing agreement permits), score, and metadata (document title, date, classification). Raw document bytes are never returned in the retrieval protocol. For highly sensitive nodes, chunk text may be replaced with a summary or an opaque reference, with the coordinator noting "relevant content exists but is not disclosable."

OWASP LLM Top 10 Mitigations

9. Governance Considerations

Federation Governance Framework

Each federation must have a formal data sharing agreement that specifies: which nodes participate, what metadata each node may expose, the agreed embedding model version, the score normalisation method, audit log sharing obligations, and the dispute resolution process when a node incorrectly declines a coordinator request.

Governance Artefacts

10. Operational Considerations

Monitoring

Service Level Objectives

11. Cost Considerations

Cost Drivers

Indicative Cost Range

12. Trade-Off Analysis

Centralised vs. Federated RAG

Architectural Tensions

13. Failure Modes

14. Regulatory Considerations

15. Reference Implementations

AWS (Multi-Region Federation)

• Local nodes: OpenSearch k-NN in each AWS region; Lambda-based retrieval API
• Coordinator: ECS Fargate service in primary region; Step Functions for fan-out
• Inter-node security: AWS PrivateLink or VPC peering + mTLS; IAM cross-account roles
• Score normalisation: Lambda function with RRF implementation

Azure (Cross-Tenant Federation)

• Local nodes: Azure AI Search per tenant; Azure Functions as retrieval API
• Coordinator: Azure Container Apps; Logic Apps for fan-out orchestration
• Inter-node security: Azure AD B2B + managed identity; Private Endpoints
• Score normalisation: Azure Function with RRF

On-Premises + Cloud Hybrid

• Local nodes: Weaviate or Qdrant on-premises + cloud nodes
• Coordinator: Kubernetes deployment with custom federation service
• Inter-node security: WireGuard VPN between on-premises and cloud; mTLS with SPIFFE/SPIRE

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Emerging — The core retrieval federation mechanics are proven; score normalisation at scale and privacy-preserving query obfuscation are active research areas; production deployments are limited to well-resourced organisations.

18. Revision History