OWASP LLM Top 10

Attackers craft inputs that override system-level instructions, hijacking model behaviour to exfiltrate data, bypass controls, or execute unintended actions. Both direct (user-supplied) and indirect (retrieved content) injection vectors apply. In 2024, a major Australian retail bank's AI customer service agent was manipulated via prompt injection to reveal internal policy thresholds by an external security researcher — the finding was disclosed to APRA under CPS 234 §24 as a near-miss information security incident.

Model-generated content is passed downstream — to browsers, shells, or APIs — without sanitisation. This enables cross-site scripting, remote-code execution, SSRF, or privilege escalation depending on the consuming system. Specific attack vectors include LLM outputs containing <script> tags passed to a browser renderer, path traversal sequences (../../../etc/passwd) inserted into file-handling pipelines, and SQL fragments (''; DROP TABLE users;--) passed directly to database query builders without parameterisation.

Adversarial data introduced during training or fine-tuning degrades model integrity, embeds backdoors, or creates systematic biases that persist invisibly across every subsequent inference.

Resource-exhaustive prompts — extremely long contexts, repetitive generation loops, or computationally expensive reasoning chains — degrade availability and drive up inference costs disproportionately. Specific attack patterns include adversarial long-context attacks (sending 190K token inputs to models with 200K context windows to maximise KV-cache pressure), recursive prompt self-expansion (where model output re-enters the context window and grows unboundedly), and repeated tool-call loops that exhaust per-minute rate limits and trigger cascading retry storms across shared AI infrastructure.

Risks introduced through third-party model weights, fine-tuning datasets, embedding providers, plugins, or SDKs with unverified provenance — any of which may carry vulnerabilities, backdoors, or malicious behaviour.

Models memorise, regurgitate, or are manipulated into disclosing PII, credentials, proprietary data, or confidential system context embedded in training corpora, system prompts, or retrieval pipelines. For Australian healthcare deployments specifically: Medicare numbers, My Health Record identifiers, and Tax File Numbers (TFNs) are Category 1 sensitive information under the Privacy Act 1988 — inadvertent AI-driven disclosure of any of these identifiers triggers mandatory notification obligations under the Notifiable Data Breaches (NDB) scheme administered by the Office of the Australian Information Commissioner (OAIC), with notification required within 30 days of becoming aware of an eligible data breach.

Plugins or tools connected to an LLM lack proper authentication, input validation, or least-privilege scoping — allowing model outputs to trigger unintended operations through an insufficiently hardened tool surface.

An LLM-powered agent is granted more permissions, capabilities, or autonomy than the task requires — enabling it to take high-impact, irreversible actions without human oversight or appropriate guardrails. This is the #1 risk for Australian financial services under APRA CPS 230 operational resilience requirements: an autonomous AI agent that can execute fund transfers, modify customer records, or submit regulatory filings without a human-in-the-loop approval gate breaches the "four-eyes principle" embedded in most Australian Financial Services Licence (AFSL) compliance frameworks and the APRA-mandated Board-level accountability model for material operational risk decisions.

Users or automated systems trust LLM outputs without appropriate scepticism or verification — particularly dangerous in regulated decisions where confidently presented hallucinations carry the same surface authority as accurate responses.

Adversaries extract proprietary model weights, fine-tuned parameters, or system prompts through repeated querying, model inversion attacks, or direct access to insufficiently protected model artefacts.