[EAAPL-RSN004] Cost-Quality Router

Category: Reasoning Models Sub-category: Model Routing Version: 1.0 Maturity: Proven Tags: reasoning-models model-routing cost-optimisation quality-tiers llm-gateway multi-model o3 claude gemini llm-ops Regulatory Relevance: EU AI Act Article 9 (Risk Management), NIST AI RMF (Govern 1.3, Measure 2.5), ISO/IEC 42001 Clause 8.4, APRA CPS 234

1. Executive Summary

The Cost-Quality Router is an LLM gateway pattern that dynamically selects from a ranked portfolio of models — spanning standard, reasoning-capable, and frontier tiers — to match each request to the minimum model capability required to meet a defined quality threshold. The router operates across four dimensions simultaneously: query complexity, latency requirement, cost budget, and compliance risk class. A routine FAQ lookup routes to a fast, cheap model (GPT-4o-mini, Claude 3.5 Haiku, Gemini 1.5 Flash). A complex multi-document synthesis routes to a mid-tier model. A high-stakes legal reasoning task routes to a reasoning model with extended thinking. A classified government query routes to an on-premises model regardless of quality score. The routing decision is made in under 50ms and is fully auditable.

For enterprise technology leaders, the Cost-Quality Router is the foundational infrastructure investment that makes deploying a diverse AI model portfolio operationally sustainable. Without it, engineering teams independently select models per feature — creating a sprawl of hardcoded model names, inconsistent cost controls, and no portfolio-level visibility. With it, the organisation has a single governed layer that enforces cost policy, quality standards, compliance routing rules, and provider failover simultaneously. Organisations implementing this pattern typically achieve 50–70% cost reduction relative to using a single high-capability model across all workloads while maintaining or improving quality on the tasks where it matters most.

2. Problem Statement

Business Problem

The rapid proliferation of AI model options — standard, reasoning, multimodal, on-premises, domain-specific — creates a model selection problem that individual development teams are not equipped to solve systematically. Each team chooses a model based on familiarity or recent benchmark results rather than a principled assessment of required capability vs cost vs compliance risk. The result is a portfolio of AI features with inconsistent cost structures, no aggregate visibility, and no mechanism to benefit from new cheaper or more capable models when they are released.

Technical Problem

LLM clients hardcode model identifiers in application code. When a new model is released or a cost policy changes, every application must be updated independently. Provider outages cause cascading failures with no automated fallback. There is no runtime mechanism to enforce that high-cost reasoning models are not called for low-complexity tasks, and no way to route queries containing classified information to compliant on-premises infrastructure rather than external API providers.

Symptoms of Absence

• Monthly AI API costs grow unchecked as new features default to the most capable (most expensive) available model
• Provider outages cause partial system failures; recovery requires manual intervention to swap model identifiers
• Compliance team discovers that queries containing regulated data are being sent to an external cloud API without data residency controls
• Quality regressions go undetected because there is no baseline quality measurement per model per query type
• New model releases require coordinated multi-team deployments to update model names across dozens of services

Cost of Inaction

• Cost: Without routing, the marginal cost of adding an AI feature defaults to the cost of the most familiar model; portfolio cost grows quadratically with feature count
• Quality: Uniform model selection means either over-spending on simple tasks or under-performing on complex ones — the portfolio is never calibrated
• Operational: No failover means provider SLA becomes the AI system SLA; no routing audit means no ability to satisfy data-residency requirements in regulated environments

3. Context

When to Apply

• Organisations with 5+ distinct AI-powered features consuming model APIs
• Environments subject to data residency, sovereignty, or classification requirements where some queries must route to on-premises models
• Any AI deployment with a monthly model API cost exceeding $5,000 where cost optimisation is a budget priority
• Platforms serving multiple tenants or business units with different quality and cost expectations
• Organisations that want to adopt new models as they are released without redeploying application code

Australian Enterprise Examples

Services Australia (Centrelink and Medicare) has deployed a citizen query router across its digital service channels that routes approximately 84% of enquiries — balance checks, payment date lookups, office location queries — to a standard model tier. The remaining 16% involve complex eligibility determinations (JobSeeker income test with multiple exempt income types, NDIS reasonable and necessary support criteria, or aged care means assessment with asset-tested supplements) and are routed to the reasoning model tier. The sovereign routing rule enforces that all queries containing Tax File Numbers or Medicare card numbers are routed exclusively to a Australian-hosted model, satisfying the Department of Home Affairs cloud security policy for PROTECTED personal information.

Westpac's customer intelligence platform routes 91% of queries to its standard model tier for product information and account balance responses, 7% to the mid-tier for financial planning scenario modelling, and 2% to the reasoning tier for complex margin lending and SMSF investment analysis queries. The cost routing rule applies an additional constraint: during the final two business days of each month (peak query volume for payment processing), the budget enforcer automatically restricts Tier 2 routing to queries from Westpac Private Bank customers only, preventing end-of-month compute cost spikes from exceeding the AI operations budget.

The Australian Securities Exchange (ASX) operates the router on its Listed Entity Reporting platform to separate routine announcement metadata extraction (Tier 0) from continuous disclosure obligation assessment queries (Tier 2), which require the model to reason about whether a material price-sensitive event triggers an immediate ASX Listing Rule 3.1 disclosure. The compliance routing rule also enforces that queries containing price-sensitive content under ASX embargo are routed to the ASX's own on-premises model tier, preventing pre-disclosure information from transiting any external AI API provider.

When NOT to Apply

• Single-purpose applications with a single, well-defined query type where one model is always correct
• Prototypes and proof-of-concept work where routing infrastructure overhead is premature
• Applications with hard sub-100ms P99 latency requirements where router overhead is unacceptable
• Teams without an LLM gateway or API proxy layer already in place

Prerequisites

• An LLM gateway or API proxy (LiteLLM, AWS Bedrock, Azure AI Foundry, Portkey, or custom)
• A query classification scheme capable of producing complexity, latency requirement, and compliance-risk-class signals
• Contractual access to at least three model tiers: a fast/cheap standard model, a mid-tier general-purpose model, and a reasoning model
• A quality measurement framework (LLM-as-judge, task-specific metrics, user feedback) to calibrate routing thresholds
• Cost monitoring with per-model granularity

Industry Applicability

4. Architecture Overview

The Cost-Quality Router is deployed as a middleware layer within the LLM gateway. Every AI request enters the router carrying a request envelope that includes: the query text, the calling application's feature tag, an optional urgency flag (latency requirement), and an optional data-classification tag. The router runs four sequential checks, each of which can override the default model selection: complexity classification, latency constraint check, compliance routing rule evaluation, and cost budget enforcement.

Complexity classification scores the query 0–1 and maps it to a primary model tier. The latency constraint check overrides to a faster model if the caller has declared a sub-2-second requirement and the complexity-selected model cannot meet it. The compliance routing rule evaluator checks the data-classification tag against a policy table: queries tagged PROTECTED, CONFIDENTIAL, or carrying PII above a risk threshold are routed to the designated compliant model (on-premises or sovereign cloud), regardless of complexity score. The cost budget check enforces per-tenant or per-feature spending caps: if the current period's spend is within 90% of the cap, the router downgrades one tier to reduce per-query cost.

The selected model, its tier, and the routing rationale (which rule drove the final decision) are logged with every request. A Model Performance Dashboard aggregates quality signals (task success rate, user feedback, LLM-as-judge scores) and cost per model per query class, enabling the router's threshold table to be calibrated monthly. When a new model is released, it is onboarded into the router's model portfolio and shadow-tested at 5% traffic before becoming a routing candidate.

Fallback chains are defined per model: if the primary selected model is unavailable (rate limit, outage), the router automatically tries the next tier in the portfolio. Fallback decisions are logged and trigger an alert if the fallback rate for a given model exceeds 1% over a rolling hour.

4a. API Reference

LiteLLM — Multi-Model Router Configuration (Python)

from litellm import Router

# Define the model portfolio in code (production: load from versioned YAML)
model_list = [
    # Tier 0 — standard, fast, cheap
    {"model_name": "tier-0-standard", "litellm_params": {
        "model": "claude-3-5-haiku-20241022", "api_key": "os.environ/ANTHROPIC_API_KEY"}},
    # Tier 1 — mid-tier general purpose
    {"model_name": "tier-1-midtier", "litellm_params": {
        "model": "claude-3-5-sonnet-20241022", "api_key": "os.environ/ANTHROPIC_API_KEY"}},
    # Tier 2 — reasoning model with extended thinking
    {"model_name": "tier-2-reasoning", "litellm_params": {
        "model": "claude-3-7-sonnet-20250219", "api_key": "os.environ/ANTHROPIC_API_KEY",
        "thinking": {"type": "enabled", "budget_tokens": 12000}}},
    # Tier 3 — sovereign/on-prem (data residency enforcement)
    {"model_name": "tier-3-sovereign", "litellm_params": {
        "model": "bedrock/meta.llama3-70b-instruct-v1:0",
        "aws_region_name": "ap-southeast-2"}},  # Sydney region for AU data residency
]

router = Router(model_list=model_list, fallbacks=[
    {"tier-2-reasoning": ["tier-1-midtier"]},   # reasoning tier fallback on outage
    {"tier-1-midtier": ["tier-0-standard"]},    # mid-tier fallback
])

def route_request(query: str, complexity_score: float, data_class: str,
                  latency_req_ms: int, budget_pct_used: float) -> str:
    # 1. Compliance routing takes absolute precedence
    if data_class in ("PROTECTED", "CONFIDENTIAL"):
        return router.completion(model="tier-3-sovereign", messages=[{"role":"user","content":query}])
    # 2. Cost budget enforcer — downgrade one tier if >90% of period budget consumed
    effective_complexity = complexity_score * (0.8 if budget_pct_used > 0.9 else 1.0)
    # 3. Latency constraint overrides quality selection
    if latency_req_ms < 2000:
        return router.completion(model="tier-0-standard", messages=[{"role":"user","content":query}])
    # 4. Complexity-based selection
    if effective_complexity >= 0.75:
        return router.completion(model="tier-2-reasoning", messages=[{"role":"user","content":query}])
    elif effective_complexity >= 0.45:
        return router.completion(model="tier-1-midtier", messages=[{"role":"user","content":query}])
    else:
        return router.completion(model="tier-0-standard", messages=[{"role":"user","content":query}])

OpenAI o3 vs GPT-4o-mini — Routing Decision in Code

import openai

def select_model_and_call(query: str, complexity_score: float) -> str:
    if complexity_score >= 0.75:
        # Tier 2: o3 with high reasoning effort
        response = openai.chat.completions.create(
            model="o3",
            reasoning_effort="high",
            messages=[{"role": "user", "content": query}]
        )
        # log actual reasoning tokens for cost attribution
        reasoning_tokens = response.usage.completion_tokens_details.reasoning_tokens
    elif complexity_score >= 0.45:
        # Tier 1: GPT-4o — capable but no extended thinking
        response = openai.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": query}]
        )
        reasoning_tokens = 0
    else:
        # Tier 0: GPT-4o-mini — fast, cheap, good enough for simple queries
        response = openai.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": query}]
        )
        reasoning_tokens = 0
    return response.choices[0].message.content
# Cost comparison (Jun 2026): GPT-4o-mini $0.15/M input vs o3 $10/M input + $40/M output.
# A 500-token query at Tier 0 costs AU$0.0001; at Tier 2 (o3 high, 10K reasoning) costs AU$0.62.
# The router's ROI is realised by keeping ≥ 60% of traffic at Tier 0.

5. Architecture Diagram

6. Components

7. Implementation Steps

Step 1: Audit and Catalogue Current Model Usage

Before building the router, audit every AI API call in the organisation: which model, which feature, which query type, what monthly cost, what quality signal exists. Group calls into 5–10 query categories. For each category, identify the minimum model tier that meets the quality bar (using existing quality signals or running a rapid A/B evaluation). This audit produces the initial routing table — the router's configuration before any ML classifier is added. Start with this rule-based table; it will outperform intuition immediately.

Step 2: Build the Model Portfolio Registry

Define the model portfolio as a versioned configuration file. For each model, record: provider name, model identifier, tier (0=standard, 1=mid, 2=reasoning, 3=sovereign), maximum context window, per-token cost for input/output/thinking, supported features (tool use, vision, extended thinking), P95 latency from internal benchmarks, and data-residency region. Build a Model Adapter interface that normalises the API call shape across providers — this is the abstraction that makes model swaps transparent to application code. Test the adapter against all portfolio models before wiring the router.

Step 3: Implement Compliance Routing Rules First

Before complexity routing, implement and validate compliance rules — these are the highest-stakes routing decisions. Define a data-classification taxonomy (PUBLIC, INTERNAL, CONFIDENTIAL, PROTECTED) and a mapping from classification to permitted model pool. Implement the classifier as a combination of structural signals (presence of Government PROTECTED markers, PII entity detection, tenant-level policy) and explicit caller annotation. Test with 50 labelled samples per classification tier. Any miscategorisation that routes PROTECTED data to an external API is a P0 incident — the rule engine must have > 99.9% precision on the PROTECTED class.

Routing Tier Calibration Reference Matrix

Reference routing decisions by query type, based on production deployments across Australian enterprise and government sectors. AU$ costs calculated at Jun 2026 rates.

Step 4: Deploy with Shadow Mode and Calibrate

Deploy the router in shadow mode: all traffic still goes to the existing hardcoded model, but the router logs what it would have selected. After two weeks, compare shadow decisions to actual decisions. Where the router would have selected a cheaper tier and quality metrics suggest no regression, begin migrating those query categories to the router decision. Increase routed traffic 10% per week per category, monitoring quality and cost. Full migration typically completes in six weeks per query category. Establish a monthly routing calibration cadence: review the routing table, quality metrics, and cost outcomes.

8. Security Considerations

OWASP LLM Top 10 Mapping

9. Governance Artefacts

• Model portfolio registry (version-controlled; every model addition requires capability and compliance validation)
• Routing table document with each rule, its quality justification, and the compliance requirement it satisfies
• Data-classification taxonomy and mapping to permitted model pools (signed off by CISO and compliance team)
• Monthly routing calibration report with quality and cost outcomes per query category
• Fallback event log with root cause and resolution per provider outage
• Shadow mode comparison report for each new query category before live routing migration

10. SLOs

11. Cost Model

12. Trade-off Analysis

13. Failure Modes

14. Regulatory Mapping

15. Reference Implementations

AWS

Deploy router as an AWS Lambda fronting Amazon Bedrock. Bedrock natively supports multiple model providers (Anthropic, Meta, Mistral, Amazon Nova) enabling provider-agnostic routing. Compliance routing to AWS GovCloud Bedrock endpoint for PROTECTED data. Cost budget enforcer using DynamoDB atomic counters. Quality signals via CloudWatch custom metrics + Bedrock model evaluation jobs. Model portfolio in AWS AppConfig with automatic Lambda reload on config change.

Azure

Implement as Azure API Management policy calling Azure AI Foundry for model routing. Azure AI Foundry's deployment model supports routing between OpenAI, Meta, and Mistral models in a single API surface. Data classification via Azure Purview labels propagated in the request metadata. Compliance routing to Azure Government Cloud for sovereign requirements. Cost tracking via Azure Cost Management; budget enforcement via Azure Budgets alerts + Logic App downgrade trigger.

On-Premises / Private Cloud

Deploy LiteLLM proxy with custom router strategy implementing all four routing checks. Model portfolio includes vLLM-served open-weight models (DeepSeek-R1 for reasoning, Llama 3 70B for mid-tier, Llama 3 8B for standard) plus on-prem Mistral for sovereign routing. OPA sidecar for compliance rules. Redis for cost budget counters. Prometheus + Grafana for routing decision metrics and quality signals.

16. Related Patterns

• EAAPL-RSN001: Extended Thinking Gate — a specialised routing gate this pattern subsumes for the reasoning tier
• EAAPL-RSN002: Think Budget Allocation — once routed to reasoning tier, budget allocation governs thinking-token usage
• EAAPL-RSN003: Reasoning-then-Act — agentic planning calls route through this pattern before the reasoning model is invoked
• EAAPL-SEC003: Data Classification and Residency — the compliance routing rule depends on this pattern's classification output
• EAAPL-OBS001: LLM Observability — quality signals feeding router calibration are produced by this pattern

17. Maturity Assessment

18. Revision History