[EAAPL-SEC002] Prompt Firewall

Category: Security / Threat Prevention Sub-category: Adversarial Input Defence Version: 1.3 Maturity: Proven Tags: prompt-injection jailbreak input-validation content-policy nlp-security classifier defence-in-depth Regulatory Relevance: APRA CPS234, EU AI Act Art. 9 & 15, OWASP LLM01, NIST AI RMF MANAGE 1.3

1. Executive Summary

The Prompt Firewall is an inline defensive layer that inspects every user input and system-constructed prompt before it reaches a large language model. It detects and blocks prompt injection attacks, jailbreak attempts, policy violations, and adversarial instructions that seek to override the model's intended behaviour or extract sensitive information.

For business stakeholders, the risk is concrete: a single successful prompt injection can cause an AI-powered application to ignore its system instructions, impersonate another user, exfiltrate data from its context window, or generate harmful content — all of which carry regulatory, reputational, and financial consequences. A prompt firewall reduces this risk to near zero for known attack patterns and significantly degrades the success rate of novel attacks through semantic analysis.

Unlike perimeter firewalls that operate on network packets, a prompt firewall operates on natural language — requiring a combination of rule-based detection (fast, deterministic, low false-positive), semantic similarity analysis (catches paraphrased attacks), and ML classifiers (catches novel attack classes). The pattern is deployed as an inline middleware stage, typically within the AI Gateway (EAAPL-SEC001), and adds 20–50ms of latency while providing a material reduction in successful prompt injection incidents.

2. Problem Statement

Business Problem

Organisations deploying AI assistants — customer service bots, internal productivity tools, code generation assistants — face an attack vector with no analogue in traditional software: natural language manipulation of the AI's behaviour. An attacker does not need to find a SQL injection vulnerability or exploit a buffer overflow. They need only craft a message that convinces the model to ignore its instructions, impersonate another user, or disclose information it should not.

High-profile incidents have demonstrated that even production LLM deployments from major vendors are vulnerable to prompt injection. The business consequences include: leakage of system prompts (containing proprietary logic or sensitive context), data exfiltration from the context window (e.g., previous conversation turns containing other users' data), generation of policy-violating content that causes regulatory exposure, and denial of service through resource-exhausting prompts.

Technical Problem

LLMs process user input and system instructions in the same channel (the prompt). Unlike a database that cleanly separates queries from data, an LLM cannot inherently distinguish between "authorised instruction" and "adversarial instruction embedded in user data." Any user-controlled text that reaches the model's context window is potentially an attack surface.

Prompt injection attacks take multiple forms: direct injection (attacker directly sends malicious instructions), indirect injection (attacker embeds malicious instructions in documents or web pages that the AI retrieves), jailbreaking (persuasion-based attempts to bypass safety training), role-play exploitation (convincing the model it is a different, unconstrained entity), and token manipulation (using special characters, encoding tricks, or unusual spacing to bypass simple pattern matching).

Symptoms

• AI application generating content that violates its stated purpose (e.g., a coding assistant generating phishing emails).
• System prompt contents appearing in model responses.
• Users reporting that the AI "acted differently" after an unusual input.
• AI application performing actions it was not instructed to perform by the application (in agentic contexts).
• Sudden spikes in content policy violations in output filtering logs.

Cost of Inaction

3. Context

When to Apply

• Any AI application that accepts user-generated text as input to an LLM.
• AI applications operating in adversarial environments (public-facing, customer-facing, or accessible by untrusted internal users).
• Agentic systems where LLMs can invoke tools, APIs, or execute code — the consequences of injection are significantly higher.
• Applications where the system prompt contains sensitive instructions, proprietary logic, or confidential context.
• Regulated use cases where policy-violating outputs carry compliance risk.

When NOT to Apply

• Fully internal, developer-only AI tools where all users are trusted and the threat model does not include insider adversaries.
• Batch processing pipelines where inputs come exclusively from trusted, validated internal sources with no user-controlled content.
• Scenarios where the latency overhead (20–50ms) is prohibitive and alternative controls (strong output filtering) provide acceptable coverage.

Prerequisites

Industry Applicability

4. Architecture Overview

The Prompt Firewall is a multi-stage detection pipeline that processes every prompt before it reaches the LLM. The pipeline architecture is designed around a fundamental principle: layered defence with increasing cost and decreasing false-positive rate at each layer. Fast, cheap checks run first; expensive, accurate checks run only when cheap checks are inconclusive.

Layer 1: Pattern Matching (Deterministic)

The first layer operates on character and token sequences. It applies a library of regular expressions and exact-match patterns derived from a constantly updated catalogue of known injection strings, jailbreak templates, and policy-violating phrases. This layer executes in microseconds and catches the vast majority of script-kiddie attacks and known jailbreak variants. The pattern library is maintained as a versioned configuration artefact, updated through a CI/CD pipeline that incorporates patterns from public jailbreak repositories (JailbreakChat, LLM Security research) and internal incident findings.

Layer 2: Semantic Analysis (Vector Similarity)

Pattern matching is defeated by paraphrasing. An attacker who knows the patterns can rephrase an injection attack to avoid any string-match. The semantic layer addresses this by embedding the input into a vector space and computing similarity against a library of known malicious embeddings. A cosine similarity threshold (typically 0.85) triggers a block. This layer catches paraphrased attacks and novel variants that share semantic intent with known attacks. It adds approximately 10–20ms using a lightweight embedding model (e.g., sentence-transformers/all-MiniLM-L6-v2 running on CPU). The embedding library is updated with new malicious examples whenever a new attack pattern is identified in the wild.

Layer 3: ML Classifier (Probabilistic)

The third layer applies a fine-tuned binary classifier trained specifically to distinguish legitimate prompts from adversarial ones. Unlike the semantic layer which measures distance to known attacks, the classifier learns decision boundaries from a labelled dataset of benign and malicious prompts — including novel attack types. This layer provides the highest accuracy but also the highest latency (30–80ms on CPU, 5–15ms on GPU). For latency-sensitive applications, this layer runs asynchronously: the request is allowed through with monitoring, but a definitive classifier decision is stored and used to update the pattern library and trigger retrospective review if the classifier scores high probability of injection.

Policy Enforcement Layer

Beyond injection detection, the firewall enforces content policies: does this input request content that violates the organisation's AI Acceptable Use Policy? This includes checks for: requests for content involving minors, attempts to obtain detailed instructions for illegal activities, requests that target specific individuals, and use-case-specific policy violations (e.g., a financial assistant being asked to produce stock tips). Policy checks use a combination of pattern matching and classifier models trained on the specific policy domain.

Allow/Deny List Management

The firewall maintains per-application allow lists (patterns that should never be blocked regardless of classifier score — e.g., legitimate security research applications) and deny lists (patterns that should always be blocked). Allow lists are critical for preventing false positives in legitimate use cases; they require governance review before addition to prevent allow list abuse.

Sanitisation Path

Not all suspicious inputs result in a block. For inputs that are ambiguous (e.g., a high pattern-match score but low semantic similarity), the firewall can sanitise: stripping suspicious instruction sequences while preserving the legitimate intent of the input. Sanitisation is logged and flagged for review to identify attack pattern evolution.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Flow

Error Flow

8. Security Considerations

Authentication & Authorisation

• The firewall service itself must be accessible only from authorised callers (AI Gateway, application middleware). mTLS or API key authentication prevents direct access.
• Pattern library and classifier model updates are authorised through a signed artefact pipeline — an attacker who can modify the pattern library can blind the firewall to specific attacks.

Secrets Management

• If the firewall uses a cloud embedding API (e.g., OpenAI Embeddings for the similarity layer), the API key must be managed per EAAPL-SEC008. Preferably, use a locally-deployed embedding model to avoid sending potentially sensitive prompt content to an external embedding provider.

Data Classification

• Prompts processed by the firewall may contain sensitive data (PII, confidential context). The firewall should not log full prompt content at INFO level; log only truncated indicators, hashes, or anonymised representations unless explicitly configured for full-content logging under a controlled data handling agreement.

Encryption

• All firewall service communication over TLS 1.3.
• Firewall event logs encrypted at rest.
• Classifier model weights stored in encrypted object storage; access audited.

False Positive Management

• False positives (blocking legitimate inputs) are a security misconfiguration, not a minor inconvenience. High false-positive rates cause users to route around the firewall or disable it. Maintain false-positive rate <0.5% of legitimate traffic.

Auditability

• Every firewall decision is logged with full reasoning: which layers triggered, what scores were returned, which pattern matched. This supports both security operations (investigating incidents) and model improvement (identifying false positives).

OWASP LLM Top 10 Coverage

9. Governance Considerations

Responsible AI

• The prompt firewall enforces the organisation's AI Acceptable Use Policy at the input layer. Policy rules must be reviewed by the AI Ethics and Governance function before deployment to ensure they do not introduce discriminatory filtering (e.g., blocking inputs in non-English languages disproportionately).

Model Risk Management

• The classifier model used in Layer 3 is itself an AI model and subject to model risk management: it must be validated on representative samples of legitimate traffic before deployment, and its false-positive and false-negative rates must be documented.

Human Approval

• ALLOW+WATCH dispositions (medium-confidence suspicious inputs that were allowed through) must be reviewed by a security analyst within 24 hours. Confirmed injections trigger classifier retraining.

Traceability

• Every block event is traceable to the specific pattern, embedding similarity score, or classifier score that triggered it. This supports appeals processes (a user who believes their input was wrongly blocked can request a review) and regulatory enquiries.

Governance Artefacts

10. Operational Considerations

Monitoring

• Real-time dashboard: block rate by layer (Pattern / Semantic / Classifier / Policy), false-positive rate (from analyst review), latency per layer, classifier confidence distribution.
• Alerting: block rate spike (>10× baseline) = possible coordinated attack; FPR spike = classifier degradation; layer unavailability = degraded defence posture.

SLOs

Logging

• Structured JSON. Mandatory fields: trace_id, disposition, layer_triggered, pattern_id (if pattern match), semantic_score, classifier_score, policy_rule_id, latency_ms, input_hash, timestamp_utc.
• Full input content logged only at AUDIT level under controlled access; standard logs contain only hash and truncated prefix.

Incident Management

• Block rate spike → automated alert to Security Operations.
• Confirmed novel injection technique → Security Operations escalates to threat intelligence team; pattern library update initiated within 4 hours.
• Classifier false-positive spike → immediate escalation to AI Platform team; temporary threshold relaxation if FPR >2%.

DR

Capacity

• Pattern matching: CPU-bound, scales linearly with rule count × request rate. 10,000 patterns at 1,000 req/s: ~2 CPU cores.
• Embedding inference: 30ms/request on single CPU core; 8 cores handles ~260 req/s; GPU (T4): ~5ms/request → 200 req/s/GPU.
• Classifier inference: similar to embedding; can be batched for throughput.

11. Cost Considerations

Cost Drivers

Scaling Risks

• Classifier inference becomes a bottleneck at high request rates if running on CPU. Provision GPU inference early.
• Embedding library grows with each new attack pattern added; ANN search latency increases. Prune stale embeddings and monitor search latency.

Optimisations

• Deploy embedding and classifier models as shared services (not per-application) to amortise GPU cost.
• Cache pattern matching results for identical inputs (hash-based deduplication) — many attackers repeat the same payload.
• Run Layer 3 classifier asynchronously for low-risk inputs to reduce synchronous path latency and allow CPU inference to be sufficient.

Indicative Cost Range

12. Trade-Off Analysis

Option Comparison

Architectural Tensions

13. Failure Modes

Cascading Failure

If the firewall fails open (allowing all traffic) during a targeted attack, the LLM's output filter (EAAPL-SEC006) becomes the last line of defence. Output filters are less effective at preventing injection (they can only catch the consequences, not the attack itself). Ensure output filtering is independently deployed and does not share failure domains with the input firewall.

14. Regulatory Considerations

15. Reference Implementations

AWS

Azure

GCP

On-Premises

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Proven

18. Revision History