[EAAPL-SEC001] AI Gateway

Category: Security / API Control Plane Sub-category: Traffic Management & Policy Enforcement Version: 2.1 Maturity: Mature Tags: api-gateway rate-limiting authentication cost-allocation circuit-breaker policy-enforcement ai-operations Regulatory Relevance: APRA CPS234, EU AI Act Art. 9 (Risk Management), ISO 42001 §6.1, NIST AI RMF GOVERN 1.2

1. Executive Summary

The AI Gateway pattern establishes a centralised, enterprise-grade control plane through which all AI traffic flows — inbound requests from applications and users, and outbound calls to model providers. It functions as the "first and last line of defence" for every AI interaction in the enterprise.

From a business perspective, the AI Gateway solves three compounding problems that emerge when AI usage scales without discipline: uncontrolled spend (teams independently acquiring model API keys lead to budget overruns with no visibility), inconsistent security posture (each team re-inventing authentication, logging, and abuse controls), and regulatory exposure (no single audit trail for AI interactions).

The gateway provides authentication and authorisation for every AI request, enforces rate limits and cost budgets per team or product, routes traffic intelligently across multiple model providers, captures structured logs for compliance, and breaks the circuit when downstream models are degraded. Organisations that deploy this pattern typically report 30–50% reduction in AI spend waste through visibility and quota enforcement, and can demonstrate AI audit trails to regulators within 24 hours of a request.

This pattern is the foundation upon which all other AI security and observability patterns depend. It should be the first pattern deployed in any enterprise AI programme.

2. Problem Statement

Business Problem

Enterprise organisations adopting AI at scale face ungoverned sprawl: dozens of teams independently calling OpenAI, Anthropic, Azure OpenAI, and other providers with individual API keys. There is no budget control, no unified audit trail, no abuse detection, and no single point where policy can be enforced. A single misconfigured application or compromised key can generate hundreds of thousands of dollars in model API spend within hours. Regulatory bodies (APRA, ASIC, EU regulators) increasingly require organisations to demonstrate comprehensive audit trails for AI-assisted decisions — an impossibility without a centralised control point.

Technical Problem

Without a gateway:

• Each application must independently implement auth, rate limiting, retry logic, and logging — creating N inconsistent implementations.
• Model provider credentials are distributed across dozens of services, dramatically increasing the blast radius of a credential leak.
• There is no circuit breaker: a degraded model provider cascades into application failures.
• Cost attribution is impossible: spend cannot be allocated to teams, products, or use cases.
• Routing logic (e.g., fallback to a cheaper model for low-complexity requests) must be duplicated across every consuming application.

Symptoms of Absence

• Unexplained spikes in model API bills.
• Security incidents involving leaked model API keys.
• Different applications enforcing different content policies, creating inconsistent user experiences.
• Inability to produce AI usage reports for compliance audits.
• Cascading application failures when a model provider has an outage.
• No capacity to enforce organisational AI usage policies (e.g., "no patient data to external models").

Cost of Inaction

3. Context

When to Apply

• Organisation has more than one team or application calling AI model APIs.
• AI model API spend exceeds $5,000/month or is forecast to.
• Organisation operates in a regulated industry (financial services, healthcare, government).
• Multiple model providers are in use or planned.
• Security team requires audit trails for AI interactions.
• AI applications are user-facing and require content policy enforcement.

When NOT to Apply

• Single-team proof-of-concept with a 90-day sunset — gateway adds operational overhead disproportionate to PoC scope.
• Fully offline/on-premises model inference where the model is a library call within the same process — a gateway adds latency without security benefit at the network boundary.
• When a cloud-native AI platform (e.g., Azure AI Studio with built-in APIM integration) already provides all required controls natively and team can accept vendor lock-in.

Prerequisites

Industry Applicability

4. Architecture Overview

The AI Gateway is deployed as a horizontally scalable reverse proxy that sits at the intersection of all AI-consuming workloads and all model provider endpoints. It is not a simple HTTP proxy — it is a stateful policy engine with its own data plane (real-time request processing) and control plane (policy configuration, key management, quota administration).

Why a dedicated gateway rather than embedding controls in each application?

The fundamental architectural reason is that cross-cutting concerns — authentication, rate limiting, cost allocation, audit logging, circuit breaking — are almost always implemented inconsistently when distributed across teams. The gateway externalises these concerns into a single, auditable, independently operated service. This mirrors the established API gateway pattern for REST/GraphQL APIs, extended with AI-specific capabilities.

Request Path Design

Inbound requests arrive from applications carrying a service identity token (mTLS client certificate or JWT). The gateway's authentication middleware validates the token against the enterprise IdP before any processing occurs. This ensures that unauthenticated requests fail fast and are never forwarded to model providers — preventing credential abuse if an internal application is compromised.

After authentication, the policy engine evaluates the request against a rule set: Does this caller have permission to use this model? Does this request exceed the caller's rate quota? Does this request carry a data classification label that prohibits forwarding to the requested external provider? Policy decisions are made in-process against an in-memory policy cache (refreshed from the policy store every 60 seconds) to keep decision latency under 1ms.

Routing and Provider Abstraction

The gateway abstracts model provider APIs behind a unified internal schema. Consuming applications call a single internal endpoint (/v1/chat/completions) regardless of whether the request will be served by GPT-4, Claude 3.7, or an on-premises Llama deployment. The routing layer maps requests to providers based on model name, caller preference, load, cost optimisation rules, and provider health. This abstraction is critical: it allows organisations to switch providers, add fallbacks, or introduce shadow routing for model evaluation without changing consuming applications.

Why circuit breaking at the gateway?

Model providers have variable availability SLAs, and LLM inference latency is orders of magnitude higher than typical microservice calls. Without a circuit breaker at the gateway, a degraded provider causes cascading timeouts across all consuming applications. The gateway's circuit breaker monitors error rates and latency per provider, opens the circuit when thresholds are breached (e.g., >10% 5xx over 60 seconds), routes traffic to the fallback provider, and attempts provider recovery with exponential backoff. This dramatically improves overall application resilience.

Cost Allocation Architecture

Each request is tagged with a cost allocation key (team, product, use-case, user) at ingress. The gateway calculates cost in real-time by multiplying token counts (extracted from the provider response) by the current pricing table (refreshed daily from a configuration store). Cost events are written to a time-series cost ledger. Budget monitors subscribe to this ledger and emit alerts or enforcement actions (soft-block, hard-block) when budgets are approached or exceeded. This gives finance teams the ability to allocate AI spend on a monthly basis without manual reconciliation.

Audit Logging

Every request and response traverses the audit logger, which writes a structured log record to an immutable audit log store (append-only, tamper-evident). The log record captures: caller identity, request timestamp, model requested, model served, token counts, cost, policy decisions made, response status, and a truncated hash of the request content (full content logging is optional and controlled by data classification). This log is the evidentiary foundation for regulatory compliance.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Flow

Error Flow

8. Security Considerations

Authentication & Authorisation

• Mutual TLS (mTLS): All inbound connections from consumer applications require a client certificate issued by the enterprise CA. This provides cryptographic identity that cannot be forged with a stolen JWT alone.
• JWT Validation: Bearer tokens carry caller identity, scope (which models are accessible), and expiry. Tokens are validated against the IdP's JWKS endpoint with a local cache (refreshed every 5 minutes). Short token lifetimes (15–60 minutes) limit the window of a compromised token.
• Service-to-Service Identity: Consumer applications authenticate as service principals, not human users. Human-facing applications should not forward end-user tokens to the gateway — the application authenticates as itself and includes user context as a claim.
• Scope-Based Model Access: JWT scopes define which model families a caller may access. A customer service application should not have scope to access GPT-4 if it only requires GPT-3.5-turbo. Principle of least privilege applies to model access.

Secrets Management

• Model provider API keys are never stored in gateway configuration files, environment variables, or source code. All credentials are retrieved at runtime from a vault (see EAAPL-SEC008).
• Gateway retrieves short-lived, dynamically generated credentials where the provider supports it (e.g., AWS Bedrock via IAM role assumption, Azure OpenAI via managed identity).
• Gateway logs never include raw API keys; only key IDs are logged for traceability.

Data Classification

• Requests carrying data classification labels above a permitted threshold for a given provider are blocked by the policy engine. For example, requests labelled CONFIDENTIAL may not be routed to external commercial model providers; only to on-premises inference endpoints.
• Data classification labels are injected by the consuming application or inferred by the prompt firewall (EAAPL-SEC005).

Encryption

• All traffic in transit uses TLS 1.3. TLS 1.0/1.1/1.2 are disabled.
• Audit logs are encrypted at rest using AES-256. Log encryption keys are managed separately from gateway operational keys.
• Request/response content stored in audit logs (if enabled) is encrypted with per-record keys, limiting the impact of a log store breach.

Auditability

• Every request generates an audit record with: caller identity, timestamp (nanosecond precision), model requested, model served, token counts, cost, policy decisions, response HTTP status, and a SHA-256 hash of the request body.
• Audit logs are written to an append-only store (S3 Object Lock, Azure Immutable Blob Storage) with a minimum 7-year retention for regulated entities.

OWASP LLM Top 10 Coverage

9. Governance Considerations

Responsible AI

• The gateway is the enforcement point for the organisation's AI Acceptable Use Policy. Policy rules in the policy store codify the AUP into enforceable controls.
• Every AI interaction is logged with sufficient context to support post-hoc review of AI-assisted decisions — a core requirement of responsible AI frameworks.

Model Risk Management

• The gateway's routing rules enforce which models may be used for which use cases. High-risk use cases (credit decisions, medical triage) can be restricted to approved, validated models only.
• Model version pinning at the gateway ensures that model updates do not reach production applications without going through the change management process.

Human Approval Gates

• The policy engine can require human approval for request types flagged as high-risk (e.g., requests to execute code, send communications, or modify records). Human approval workflows are triggered via an integration with the organisation's ITSM platform.

Policy Management

• AI usage policies are maintained as code (OPA Rego or Cedar policies) in a version-controlled repository. Changes undergo PR review, automated policy testing, and staged rollout through gateway environments (dev → staging → production).

Traceability

• Every policy decision is logged with the rule ID that triggered it, enabling governance teams to audit which policies are most frequently triggered, identify policy gaps, and demonstrate regulatory compliance.

Governance Artefacts

10. Operational Considerations

Monitoring

• Gateway metrics must be collected at sub-second granularity: request rate, error rate, p50/p95/p99 latency per provider, token throughput, quota utilisation per caller, circuit breaker state, and cost rate.
• Dashboards provide both real-time operational view (SRE) and 30-day trend view (governance).

SLOs

Logging

• Structured JSON logs. Mandatory fields: trace_id, span_id, caller_id, model_requested, model_served, request_tokens, response_tokens, cost_usd, policy_decision, http_status, latency_ms, timestamp_utc.
• Log level INFO for all requests; WARN for policy violations; ERROR for auth failures and circuit breaker events; AUDIT for all request/response pairs (separate immutable log stream).

Incident Management

• P1: Gateway unavailable — all AI workloads impacted. Pager alert to AI Platform SRE + escalation to Architecture owner within 5 minutes.
• P2: Provider circuit open with no healthy fallback. Pager alert; initiate fallback provider activation.
• P3: Sustained rate of policy violations (>1% of requests). Alert to Security Operations for investigation.

DR

Capacity

• Gateway is stateless in the data plane (policy decisions made against in-process cache). Scale horizontally with demand.
• Redis quota store: size for peak token rate × TTL. At 10M tokens/minute with 1-minute rolling window: ~10M counters × 20 bytes = ~200MB — comfortably within Redis memory limits.
• Provision for 3× normal peak to absorb burst without autoscaling lag.

11. Cost Considerations

Cost Drivers

Scaling Risks

• Audit log storage grows unboundedly. Implement tiered storage (hot → warm → cold → archive) with automated lifecycle policies.
• Redis memory pressure at extreme token volumes. Use token bucket algorithm with decay to limit state size.

Optimisations

• Cache policy decisions for stable caller+model combinations (1-minute TTL) to avoid OPA evaluation on every request.
• Use spot/preemptible instances for non-stateful gateway replicas (failover to on-demand automatically).
• Compress audit logs before writing to object storage (LZ4/Zstandard): typical 70–80% compression ratio on structured JSON.

Indicative Cost Range

These figures cover gateway infrastructure only. Model provider API costs are the dominant expenditure and are not included.

12. Trade-Off Analysis

Option Comparison

Architectural Tensions

13. Failure Modes

Cascading Failure Scenarios

Scenario 1: Vault + Redis simultaneous failure If both Vault (credential store) and Redis (quota store) fail simultaneously, the gateway cannot retrieve credentials AND cannot enforce quotas. The gateway must fail closed (return 503) — accepting quota bypass (fail-open) while Vault is down would allow unlimited uncredentialed requests to reach providers once credentials are cached. Mitigation: Vault and Redis must be deployed on independent infrastructure with no shared failure domain.

Scenario 2: Policy store becomes unavailable during a security incident If the policy store becomes unavailable at the same time as a security event requiring policy update (e.g., a compromised caller key), the gateway will continue serving the last cached policy. Mitigation: emergency policy override API that writes directly to the in-memory cache on each gateway instance; secured with break-glass credentials.

14. Regulatory Considerations

15. Reference Implementations

AWS

Azure

GCP

On-Premises

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Mature

18. Revision History