[EAAPL-MCP003] Multi-Server Orchestration

Category: MCP Sub-category: Orchestration Version: 1.0 Maturity: Emerging Tags: mcp, multi-server, orchestration, capability-composition, workflow, tool-chaining, context-management, fan-out Regulatory Relevance: EU AI Act Art. 9/15, APRA CPS 234, ISO/IEC 42001 §8.4, NIST AI RMF (GOVERN 1.6, MANAGE 2.4), Privacy Act 1988 (Cth)

1. Executive Summary

The MCP Multi-Server Orchestration Pattern describes how an AI model or agent client coordinates capability invocations across two or more MCP servers simultaneously — composing tools, resources, and prompts from independent servers to fulfil complex, multi-step workflows that no single server can complete in isolation. The pattern covers connection management (maintaining concurrent sessions to N servers), capability aggregation (presenting a unified tool surface to the model), context threading (passing relevant results from one server's tool into another server's tool invocation), and workflow observability (tracing a logical workflow across multiple server response streams).

For CIO/CTO audiences: as enterprise AI capability portfolios mature, individual MCP servers become specialised — a data retrieval server, a code execution server, a document generation server, an approval routing server. Real business workflows span all of these. Multi-server orchestration is what turns a collection of independently useful servers into a coherent AI workflow platform. The architectural decisions made here — how context is shared across servers, how partial failures are handled, how cross-server workflows are audited — determine whether complex AI workflows are reliable, governable enterprise processes or opaque black-box chains that fail in ways operators cannot diagnose or reproduce.

2. Problem Statement

Business Problem

Enterprise AI workflows routinely require capabilities that span multiple data domains, execution environments, and backend systems. A contract review workflow needs a document retrieval server, a compliance rules server, and a document annotation server. A customer support resolution workflow needs a CRM server, a knowledge base server, and a ticketing server. Without a structured orchestration pattern, these multi-server workflows are implemented ad hoc — coupling tool selection logic into prompts, hard-coding server connection sequences, and producing workflows that break silently when any one server is degraded.

Technical Problem

The MCP protocol defines how a client connects to a single server; it does not natively define how a client coordinates across multiple servers simultaneously. Without a structured orchestration layer, context passing between server invocations is uncontrolled (the model's context window becomes the only integration bus), connection lifecycle management for N concurrent server sessions is fragmented, and there is no transaction boundary or compensation mechanism when a multi-step workflow fails partway through. Distributed tracing across multiple server audit logs is impossible without a shared correlation identifier.

Symptoms of Absence

• AI workflows that require capabilities from two servers fail because the client only maintains a connection to one server at a time
• Intermediate tool results from server A are passed to server B by pasting them into the model's prompt — losing structure, type safety, and audit integrity
• A workflow that completes server A invocations but fails on server B leaves no compensation record; downstream systems are in an inconsistent state
• Distributed traces for a multi-step workflow span multiple disjoint server audit logs with no shared identifier linking them
• The model's context window fills with intermediate tool results from early workflow steps, degrading reasoning quality in later steps

Cost of Inaction

• Security: Unstructured context passing between servers transmits data across trust boundaries without explicit access control checks at each boundary
• Compliance: Multi-server workflow audit trails are fragmented across server logs with no unified workflow-level record
• Operational: Partial workflow failures produce inconsistent system state with no automated detection or compensation
• Engineering: Each new multi-server workflow is a custom implementation; common orchestration concerns (fan-out, result aggregation, retry) are reinvented per workflow

3. Context

When to Apply

• A single AI workflow requires capabilities from two or more independent MCP servers
• The workflow involves sequential steps where the output of one tool invocation is the input to the next (tool chaining)
• Fan-out patterns are needed: the same input must be sent to multiple servers in parallel and results aggregated
• The workflow has business transaction semantics — partial completion must be detectable and compensatable
• Cross-server workflow observability (distributed tracing) is required for debugging and compliance

When NOT to Apply

• All required capabilities are available on a single MCP server (no orchestration needed)
• The workflow is a single-hop retrieval with no chaining — use EAAPL-MCP001 directly
• The orchestration complexity exceeds the workflow complexity; for simple two-step workflows, direct sequential server calls in application code may be more maintainable than a full orchestration layer

Prerequisites

• Each participating MCP server is designed per EAAPL-MCP001 and is independently stable
• An MCP Gateway (EAAPL-MCP002) is preferred as the single entry point for all server connections
• A distributed tracing infrastructure capable of receiving spans with custom correlation IDs
• A workflow state store (durable, not ephemeral) for tracking multi-step workflow progress
• Clear definition of compensating actions for each workflow step that modifies state

Industry Applicability

4. Architecture Overview

Multi-server orchestration is implemented in the orchestration client layer — the component (typically a long-running agent or workflow engine) that maintains concurrent MCP sessions to multiple servers and coordinates invocations across them. This client layer is distinct from the AI model itself; the model issues tool calls, but the orchestration client is responsible for routing those calls to the correct server, managing connection state, and threading results between steps.

Connection management requires each server connection to be independently lifecycle-managed. The orchestration client maintains a connection pool keyed by server identity, with independent health monitoring for each connection. Server connections are established at workflow start (not lazily at first invocation) to surface connection failures before the workflow commits to a multi-step execution. If any required server is unavailable at workflow start, the workflow fails fast rather than discovering the unavailability mid-execution.

Context threading is the mechanism by which structured results from one server are passed as structured inputs to another. The orchestration client is responsible for this translation — it extracts the relevant fields from a tool result, validates them against the destination tool's JSON Schema, and constructs the destination invocation. The model's context window should contain only the semantic state of the workflow (what was decided, not the raw data that was retrieved), not the full content of intermediate tool results. This keeps context window utilisation efficient and prevents sensitive intermediate data from unnecessarily persisting in the model's active context.

Workflow observability is achieved by injecting a shared workflow correlation ID into every tool call across all participating servers as a structured context field. When all servers write their tool invocation audit records with this common ID, a workflow-level trace can be reconstructed by querying across server logs. The orchestration client also maintains a workflow state document — a durable record of completed steps, pending steps, and the semantic outcome of each — that serves as the single source of truth for workflow progress and the basis for compensation if a later step fails.

Wire Protocol Detail

Multi-server orchestration introduces two wire-level concerns that single-server MCP does not: correlation ID injection across independent JSON-RPC sessions, and fan-out request multiplexing where the orchestrator sends concurrent tool calls to multiple servers. The following examples show the exact message shapes.

Correlation ID injection via _meta — every tool call across all servers in a workflow carries the same workflow-scoped UUID in the _meta extension field:

// Orchestrator → Server A (step 1: fetch customer data)
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"get_customer","arguments":{"customer_id":"AU-98432"},"_meta":{"correlationId":"wf-550e8400-e29b-41d4-a716-446655440001","workflowStep":1,"workflowType":"contract_review"}}}

// Orchestrator → Server B (step 2: fetch compliance rules — concurrent with step 1 if independent)
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"get_compliance_rules","arguments":{"jurisdiction":"AU","product_type":"mortgage"},"_meta":{"correlationId":"wf-550e8400-e29b-41d4-a716-446655440001","workflowStep":2,"workflowType":"contract_review"}}}

The _meta object is the MCP spec's extension point for non-protocol metadata. The correlationId field is not interpreted by MCP servers — it is stored verbatim in the audit log and used by the Workflow Audit Aggregator to reconstruct the full workflow trace across disjoint server logs.

Context threading between steps — the orchestrator extracts structured fields from step 1's result and maps them to step 3's arguments (never passing raw result strings):

// Server A → Orchestrator: step 1 result
{"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"{\"customerId\":\"AU-98432\",\"riskTier\":\"high\",\"jurisdiction\":\"NSW\"}"}],"isError":false}}

// Orchestrator → Server C (step 3: uses extracted riskTier from step 1 + rules from step 2)
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"assess_contract_risk","arguments":{"customer_risk_tier":"high","applicable_rules":["NCCP_s131","APRA_APS220_clause14"],"document_id":"DOC-20240614-001"},"_meta":{"correlationId":"wf-550e8400-e29b-41d4-a716-446655440001","workflowStep":3,"workflowType":"contract_review"}}}

Partial workflow failure — when step 3 fails, the orchestrator must record the failure and invoke compensation against step 1's side effects using a structured error result:

// Server C → Orchestrator: step 3 execution failure
{"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"Risk assessment service unavailable"}],"isError":true}}

// Orchestrator → Server A: compensation call (reverses step 1's document lock)
{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"release_document_lock","arguments":{"document_id":"DOC-20240614-001","reason":"workflow_compensation","original_correlation_id":"wf-550e8400-e29b-41d4-a716-446655440001"},"_meta":{"correlationId":"wf-550e8400-e29b-41d4-a716-446655440001","workflowStep":"compensation-1","workflowType":"contract_review"}}}

Note that isError: true in the result object (not a JSON-RPC error object) is the MCP-specified mechanism for a tool that executed successfully at the protocol level but returned an application-level failure. The orchestrator must check result.isError on every tool response — not just the presence/absence of a top-level error field.

5. Architecture Diagram

6. Components

7. Implementation Steps

Step 1: Define the Workflow Graph

Before implementation, produce a formal workflow graph: each node is a tool invocation (specifying the target server, tool name, and argument mapping), each edge is a data dependency (output field X from step N feeds input field Y of step M). Identify which steps can run in parallel (fan-out nodes with no data dependency between them) and which must be sequential. Define the compensating action for each step that writes state — if the workflow fails after step N, what does the rollback of step N look like? This graph is the governance artefact that security and architecture review; it is not implicit in code.

Step 2: Establish and Validate Connections

At workflow startup, the orchestration client establishes MCP sessions to all servers in the workflow graph and validates that each server exposes the expected tools with compatible schemas. Do not proceed to execution if any required server is unavailable or if a tool's schema has changed in a breaking way (argument type change, removal of a required field). Schema compatibility checking at workflow start catches integration regressions before they produce partial-completion failures mid-workflow. Log the server capability snapshot taken at workflow start as part of the workflow audit record.

Step 3: Execute with Context Threading

Execute the workflow graph following the defined topology (respecting data dependencies; parallelising independent nodes). For each step, construct the tool invocation by extracting the required fields from the workflow state store (not from the model's raw output) and validating against the target tool's JSON Schema. Inject the workflow correlation ID as a structured context field in every invocation. Write the structured result of each step to the workflow state store immediately upon receipt — before the model is informed of the result and before the next step begins.

Step 4: Implement Compensation and Finalisation

Register a compensation handler for each state-modifying step before execution begins. If any step returns an error or times out, the orchestration client pauses the workflow, records the failure in the workflow state store, and executes compensation handlers for all previously completed state-modifying steps in reverse order. On successful workflow completion, write a workflow completion record to the audit log referencing all step correlation IDs. This final record is the attestation that the complete workflow executed as designed — the basis for compliance and audit queries.

AU Compliance Note — APRA CPS 234 Attachment C & APS 330: Multi-server orchestration is a specific audit challenge because a single logical AI action spans multiple independent server audit logs. APRA CPS 234 Attachment C requires that records of access to information assets be "complete and accurate" — in the context of a multi-server workflow, this requires the ability to reconstruct the full decision chain from the distributed logs. The shared correlationId injected into every tool call (see Wire Protocol Detail above) is the technical mechanism that satisfies this requirement: an APRA examiner or incident investigator can query all server audit logs for a single correlationId and reconstruct the complete workflow execution. Implement a Workflow Audit Aggregator query that can produce a human-readable workflow timeline from the distributed logs — this aggregation view is the artefact you present to APRA under Attachment C testing. Retain workflow execution records (including compensation records) for 7 years per APS 330.

8. Security Considerations

OWASP LLM Top 10 Mapping

Additional Controls

• Use a distinct service account identity per workflow type, not a single shared orchestration client identity, so that per-workflow access can be revoked independently
• Enforce a maximum workflow execution time (wall-clock timeout) at the orchestration client to prevent indefinitely running workflows holding open server connections
• Never persist raw intermediate tool results to the workflow state store without evaluating their data classification — apply field-level encryption to sensitive intermediate values at rest
• Implement a workflow instance kill-switch in the administrative API: operators must be able to terminate any running workflow instance immediately

9. Governance Artefacts

• Workflow Graph Definition — formal specification of each workflow: steps, data dependencies, target servers, and compensating actions; version-controlled
• Workflow Execution Audit Log — per-workflow-instance record of all steps, tool invocations (with correlation IDs), results, and completion status
• Server Capability Compatibility Record — snapshot of each server's capability schema taken at workflow start; retained for post-incident analysis
• Compensation Execution Record — log of every compensation action triggered, including trigger reason, compensation result, and operator notification
• Workflow Performance Baseline — documented p50/p95/p99 latency per workflow type, used to detect workflow regressions

10. SLOs

11. Cost Model

12. Trade-off Analysis

13. Failure Modes

14. Regulatory Mapping

15. Reference Implementations

AWS

Implement the orchestration client as an AWS Step Functions state machine, with each state invoking an AWS Lambda function that calls the appropriate MCP server via HTTPS. Use DynamoDB as the workflow state store with DynamoDB Streams for event-driven compensation triggers. Inject the Step Functions execution ARN as the workflow correlation ID across all server invocations. Use AWS X-Ray for distributed tracing, with custom segments for each MCP tool invocation annotated with the correlation ID.

Azure

Implement using Azure Durable Functions (fan-out/fan-in orchestration pattern) with each activity function calling an MCP server via the MCP client SDK. Use Azure Table Storage or Cosmos DB for workflow state (Durable Functions manages this natively). Use Azure Monitor Application Insights distributed tracing with a custom operation ID mapped to the workflow correlation ID. Deploy the orchestration function app in the same Azure region as the MCP servers to minimise inter-step latency.

On-Premises / Self-Hosted

Implement using Temporal.io for durable workflow orchestration — Temporal provides built-in compensation (saga pattern), workflow state persistence, and retry policies. Each Temporal activity is an MCP tool invocation. Use Jaeger for distributed tracing, injecting the Temporal workflow ID as the correlation ID. Deploy Temporal Server on Kubernetes alongside the MCP server fleet; use the same Istio service mesh for mTLS between the orchestration worker, Temporal server, and MCP servers.

16. Related Patterns

• EAAPL-MCP001: MCP Server Design — each server in the orchestrated fleet must be individually designed per this pattern
• EAAPL-MCP002: MCP Gateway — the gateway provides the unified entry point and policy enforcement for all servers in the orchestrated fleet
• EAAPL-MCP005: Stateful MCP Sessions — session state management for long-running orchestrated workflows
• EAAPL-AGT001: Agentic Workflow Orchestration — parent pattern for AI workflow orchestration of which MCP multi-server orchestration is a specialisation
• EAAPL-AGT007: Agent Checkpoint and Recovery — checkpoint and compensation patterns that apply at the orchestration level

17. Maturity Assessment

18. Revision History