[EAAPL-PLT003] Model Routing

Category: Platform Engineering Sub-category: Traffic Management Version: 1.2 Maturity: Proven Tags: model-routing, intelligent-routing, cost-based-routing, latency-routing, capability-routing, shadow-routing, a-b-routing, fallback, routing-rules-as-code Regulatory Relevance: EU AI Act Article 9 (Risk Management), ISO 42001, NIST AI RMF MAP 2.1

1. Executive Summary

The Model Routing pattern establishes intelligent, policy-driven dispatch of AI inference requests to the optimal model from a pool of candidates. As organisations operate multiple model providers and tiers—frontier models for complex reasoning, mid-tier models for standard tasks, specialist models for domain-specific workloads—the routing layer translates business intent (minimise cost, maximise quality, meet latency SLO) into per-request model selection decisions without burdening product teams with this logic.

The commercial impact is significant: organisations that implement tiered routing consistently report 30–50% reduction in model API spend by directing simple tasks to cheaper models while reserving frontier compute for genuinely complex requests. Additionally, shadow routing enables risk-free model evaluation in production traffic, and fallback routing maintains availability when individual providers degrade. Routing rules expressed as code integrate with GitOps workflows, giving governance teams an auditable, reviewable change process for every routing policy change.

2. Problem Statement

Business Problem

Organisations pay frontier model prices for tasks that could be handled by models costing 10–20× less. There is no systematic mechanism to evaluate new model versions without exposing production traffic to risk. When a model provider has an outage, AI features fail rather than failing over to an available alternative.

Technical Problem

Routing logic is hardcoded in product team applications: each team selects a specific model endpoint and implements its own fallback logic. When routing strategy needs to change (e.g., switch primary model, adjust fallback order, enable cost-based routing), each team must make independent code changes. There is no A/B framework for comparing model quality systematically.

Symptoms

• 100% of AI requests going to the single most expensive model regardless of task complexity
• New model evaluation requiring full production deployment with rollback risk
• Model provider outage causing complete AI feature failure rather than graceful failover
• No mechanism to compare quality of two models on the same production traffic
• Teams spending engineering time implementing and maintaining per-team fallback logic

Cost of Inaction

• Unnecessary model API costs of 30–50% above optimal routing
• Model evaluation cycles of 4–8 weeks due to lack of production traffic comparison tooling
• Provider outage MTTR of hours instead of minutes due to hardcoded model selection
• Inability to demonstrate model governance to auditors (no audit trail of routing decisions)

3. Context

When to Apply

• Organisation operates ≥2 model providers or model tiers simultaneously
• Cost optimisation of AI spend is a priority
• Availability requirements demand provider failover capability
• Model evaluation and comparison is a recurring operational need
• Platform team centralises model access (see EAAPL-PLT001)

When NOT to Apply

• Single model, single provider with no plans for multi-provider: routing overhead not warranted
• Models are fundamentally incompatible in output format such that failover would break consuming applications
• Ultra-low latency requirements (<100ms total) where routing overhead is prohibitive (use direct integration)

Prerequisites

• AI API Gateway (EAAPL-PLT002) as the host for routing logic
• Model Registry with capability cards per model (PLT001 Layer 2)
• Multiple model provider credentials managed in Secrets Manager
• Observability infrastructure for routing decision logging and model performance metrics
• Response schema normalisation across providers (or application tolerance for schema variation)

Industry Applicability

4. Architecture Overview

The Model Routing layer sits within or immediately behind the AI API Gateway and executes per-request model selection before the upstream proxy forwards the call. The routing decision is deterministic given the same input context and routing configuration, making it reproducible and auditable. The routing configuration is stored as code in a Git repository, enabling GitOps workflows for policy changes.

Intent Classification is the first stage of routing logic. The incoming request carries signals that inform routing: the declared use case tag in the request metadata (e.g., use-case: summarisation), the consumer's team namespace (which may have team-level routing overrides), the estimated complexity of the request (derived from prompt length, presence of structured data, declared reasoning requirement), and any explicit model hint from the consumer (which is subject to policy gating). Intent classification can be as simple as a rule lookup against the use-case tag or as sophisticated as a lightweight classifier that scores request complexity in <10ms.

Routing Strategy Evaluation applies the configured strategy for the consumer/use-case combination. Four primary strategies are defined:

Cost-based routing assigns a cost tier to each request (low/medium/high) based on complexity signals and routes to the cheapest model within that tier that meets the quality threshold. Cost tiers map to model families: low-cost (GPT-4o-mini, Claude Haiku, Gemini Flash), mid-cost (GPT-4o, Claude Sonnet), high-cost (o1, Claude Opus, Gemini Ultra). The quality threshold per tier is expressed as a minimum benchmark score on the organisation's evaluation dataset.

Latency-based routing selects the model with the lowest current P90 latency from real-time metrics. This is particularly valuable for interactive user-facing features where model quality differences are marginal but latency differences are perceived. The latency metric is maintained as a sliding 5-minute window per provider endpoint.

Capability-based routing matches the request's declared requirements against model capability cards in the registry. A request requiring 128K+ context routes only to models with sufficient context windows; a request requiring tool use routes only to models with function-calling capability; a request requiring structured JSON output routes to models with reliable JSON mode. Capability routing is essentially a filter, often combined with cost or latency routing for final selection.

Fallback routing defines an ordered preference list for a given model alias. When the primary model's circuit breaker is open or the provider returns persistent errors, the router advances to the next candidate. The fallback chain is explicit and version-controlled, not implicit.

A/B and Shadow Routing are layered on top of the primary strategy. A/B routing sends a configurable percentage of traffic to a candidate model, comparing outputs against the primary on the organisation's quality metrics. Shadow routing duplicates requests to a candidate model asynchronously without serving its response to the consumer; this enables zero-risk production traffic evaluation. Both mechanisms write routing experiment metadata to the Evaluation Framework (EAAPL-PLT008) for analysis.

Circuit Breaker Integration makes routing resilient. Each model endpoint has an associated circuit breaker tracking success rate and latency over a rolling window. When a circuit opens, the router excludes that endpoint from selection for the duration of the open window (configurable, typically 60 seconds). After the open window, a half-open state tests with a single request. This means the router inherently implements provider failover without a separate failover mechanism.

Routing Rules as Code is a first-class governance principle. All routing configuration—strategy assignments per use case and consumer, fallback chains, A/B experiment configurations, capability requirements, cost tier thresholds—is expressed in a structured configuration format (YAML/JSON) stored in the platform's Git repository. Changes go through pull request review with platform team approval and are applied to the routing engine via a configuration deployment pipeline. Every routing configuration version is recorded in the audit log alongside the routing decisions it produced.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Flow — Cost-Based Routing Request

Error Flow

8. Security Considerations

Authentication and Authorisation

• Model selection may not be manipulated by consumer input beyond the declared use-case tag; raw model names in consumer requests are validated against authorised models for that consumer
• Team-level routing overrides require platform team approval; they are stored in the version-controlled routing rules, not consumer-controllable at request time

Secrets Management

• Model provider credentials for each endpoint are retrieved from Secrets Manager at routing decision time; credentials are not embedded in routing rules
• Shadow routing uses separate credentials with read-only scoping where possible to prevent shadow model being used for mutations

Data Classification and Encryption

• Routing decisions involving RESTRICTED or CONFIDENTIAL data are logged with the classification label for audit trail completeness
• Shadow requests must be subject to the same data classification and policy enforcement as primary requests

Auditability

• Every routing decision is logged with: strategy applied, candidates considered, circuit breaker states, selected endpoint, reason code, any experiment configuration active
• Routing configuration changes are version-controlled and auditable as Git commits with author, timestamp, and review record

OWASP LLM Top 10 Controls

9. Governance Considerations

Responsible AI

• Routing rules must not route high-risk AI use cases to models without a completed Model Risk Card
• A/B experiments involving high-risk use cases require explicit Governance Board approval before activation
• Shadow routing results feed into model evaluation decisions that are recorded in the Evaluation Framework

Model Risk Management

• The routing fallback chain defines the approved substitution hierarchy; arbitrary model substitution is not permitted
• When a new model is added to the registry and routing rules, a Model Risk Card delta review is required comparing the new model to existing candidates
• Routing telemetry (which model served which volume of requests) is a key input to the quarterly model risk review

Governance Artefacts

10. Operational Considerations

Monitoring

SLOs

Logging

• Routing decisions logged as structured JSON with correlation to the gateway request ID
• Circuit breaker state transitions logged separately for operational analysis
• A/B experiment decisions include experiment ID and variant for analysis join

Incident Response

Disaster Recovery

11. Cost Considerations

Cost Drivers

Optimisations

• Most valuable optimisation: aggressive Tier 3 routing for high-volume, low-complexity tasks (summarisation, classification, entity extraction)
• Intent classifier should be rule-based for speed (latency budget <5ms) unless complexity estimation materially improves routing quality
• Cache routing decisions for identical consumer + use-case combinations with short TTL (1 minute) to reduce routing computation

Indicative Cost Range

12. Trade-Off Analysis

Routing Strategy Options

Intent Classification Options

Architectural Tensions

13. Failure Modes

Cascading Scenario

• Mass circuit opening storm: Under a broad cloud provider degradation, multiple circuits open simultaneously. The router falls back to the next tier for all requests. If the fallback tier is also degraded (same cloud region), the cascade proceeds through all fallback candidates and the router returns 503 for all requests. Mitigation: fallback chains must span cloud providers or include on-premises/alternative-region endpoints.

14. Regulatory Considerations

EU AI Act Article 9

• Routing decisions must be recorded to demonstrate that the risk management system controls which models process which use cases; the routing audit log satisfies this requirement
• High-risk AI systems must not be subject to automatic fallback to lower-quality or unapproved models without human oversight configuration

NIST AI RMF MAP 2.1

• The routing configuration explicitly documents the intended deployment context for each model, satisfying MAP 2.1's requirement to document AI deployment context

Audit and Record-Keeping

• Routing decision logs must be retained for the same period as the AI system's operational records (typically 7 years for regulated decisions)
• Routing configuration Git history constitutes an auditable record of every routing policy change with author and approval

15. Reference Implementations

AWS

Azure

GCP

On-Premises

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Proven Model routing is production-proven across dozens of enterprise deployments. LiteLLM and Kong AI Gateway provide mature implementations. ML-based intent classification is still an emerging practice; rule-based routing is the proven approach.

Scoring Matrix

18. Revision History