[EAAPL-GOV008] AI Incident Management

Category: Governance / Operational Resilience Sub-category: AI-Specific Incident Response Version: 1.1 Maturity: Proven Tags: incident-management, AI-incidents, hallucination, bias-incident, security-incident, MTTD, MTTR, regulatory-notification Regulatory Relevance: APRA CPS230 §43–§46, APRA CPS234 §36–§37, EU AI Act Article 72, Privacy Act APP 11, NIST AI RMF MANAGE 3.1

1. Executive Summary

The AI Incident Management pattern implements a specialised incident management lifecycle for AI system failures. It recognises that AI incidents have fundamentally different characteristics from conventional software incidents: failure modes include hallucination, statistical bias drift, adversarial attack, data poisoning, and emergent behaviour—none of which appear in standard ITIL incident taxonomies and none of which standard ITSM runbooks address.

The pattern extends enterprise incident management with an AI-specific incident taxonomy, AI-specific MTTD/MTTR targets, AI-specialist escalation paths, and regulatory notification workflows triggered by specific incident categories. It integrates with the AI Audit Trail (GOV007) for forensic evidence, the Model Bias Detection pipeline (GOV006) for bias incident detection, and the AI Policy Enforcement layer (GOV004) for security incident evidence.

For CIOs and CTOs, the critical strategic outcome is regulatory compliance with notification obligations. APRA CPS230 §43 requires notification of material incidents within 72 hours. The EU AI Act Article 72 requires notification of serious incidents to market surveillance authorities. Without a structured AI incident management process, organisations will miss notification windows—a governance failure that compounds the original incident.

2. Problem Statement

Business Problem

AI-specific failures are not recognised, categorised, or escalated appropriately through standard incident management processes. Hallucination incidents are logged as "system behaved unexpectedly" with no understanding of systemic risk. Bias incidents are not recognised as incidents at all. Regulatory notification obligations are missed because the incident's regulatory significance is not assessed.

Technical Problem

Standard ITSM tools (ServiceNow, Jira Service Management) do not have AI-specific incident categories, AI-specific root cause codes, or AI-specific resolution actions. AI incidents require forensic capability (audit trail queries), model behaviour analysis (fairness metrics, output sampling), and often model rollback or scope restriction — actions not in standard incident runbooks.

Symptoms

• AI failures treated as "application bugs" with standard P3 priority regardless of customer impact
• No post-incident review for AI-specific contributing factors
• Regulatory notification obligations missed for privacy breaches via AI disclosure
• Bias incidents detected by media or regulators, not internal monitoring
• Model rollback performed without documented incident process
• No AI incident trend analysis informing model risk management

Cost of Inaction

• Regulatory: APRA CPS234 §36: failure to notify APRA of material security incident within 72 hours. EU AI Act Article 72: failure to notify of serious AI incidents
• Legal: Undocumented AI incidents creating liability for unaddressed harms
• Reputational: Reactive, unstructured response to public AI failures amplifies reputational damage
• Operational: No learning from AI incidents; same failure modes recur

3. Context

When to Apply

• Any enterprise operating AI systems in production with customer, financial, or regulatory exposure
• APRA-regulated entities (mandatory for material operational incidents per CPS230)
• EU market participants operating high-risk AI (mandatory notification per EU AI Act Article 72)
• Any organisation with stated responsible AI commitments (accountability requires incident response)

When NOT to Apply

• Internal AI tools with no customer impact and no regulatory exposure (standard ITSM applies)
• Development and testing environments (internal development incidents use standard dev processes)

Prerequisites

• AI Audit Trail (GOV007) operational — forensic evidence source
• AI Model Register (GOV001) — MRID required for impact scoping
• AI Policy Enforcement (GOV004) — security incident evidence
• Enterprise ITSM platform — AI-specific incident taxonomy integrated into existing system

Industry Applicability

4. Architecture Overview

The AI Incident Management pattern extends enterprise ITSM with three AI-specific architectural additions: an AI incident taxonomy that integrates with detection sources, an AI incident playbook library that guides responders through AI-specific resolution actions, and an automated regulatory notification workflow triggered by incident classification.

AI Incident Taxonomy. The pattern defines six primary AI incident categories that do not exist in standard ITSM taxonomies:

Hallucination Incident: An AI system generates factually incorrect, fabricated, or internally inconsistent output. Severity is determined by whether the output was customer-facing, whether it could cause direct harm (medical advice, legal advice, financial advice), and whether it was acted upon before detection. Root cause investigation must determine: was this a single anomalous output or a systematic pattern? Is the hallucination triggerable by specific input patterns (vulnerability)?

Bias Incident: Statistical evidence of discriminatory model behaviour. Detected via GOV006 bias detection alerts or by external complaint/report. Severity based on: protected attribute affected, decision type (consequential vs non-consequential), affected population size, duration of bias operation. Root cause categories: training data skew, feedback loop formation, distribution shift, feature correlation with protected attribute.

Security Incident (AI-specific): Adversarial attack on AI system (prompt injection, model inversion, membership inference, model extraction). Differentiated from conventional security incidents by the AI-specific attack vector and the potential for model-layer compromise that application-layer security controls do not detect.

Availability Incident: AI system unable to provide service within SLO. While similar to conventional availability incidents, AI availability failures have distinctive characteristics: model serving infrastructure failure vs. model quality degradation (model appears available but is producing degraded outputs). The latter is an AI-specific availability failure that standard health checks miss.

Data Poisoning Incident: Evidence or strong suspicion that training data was tampered with to produce targeted model behaviour. Requires forensic investigation of training data provenance. Rarest incident type but most severe consequence.

Privacy Disclosure Incident: AI system discloses personal information that should not be accessible, via membership inference (confirming someone's data was in the training set), training data memorisation (reproducing personal information from training data), or policy enforcement failure. Triggers Privacy Act notifiable data breach assessment.

Graduated Severity Model. AI incidents are assigned four severity levels that determine response times and notification obligations:

• P1 (Critical): Imminent or actual harm to individuals; regulatory notification likely required; immediate containment required
• P2 (High): Significant risk of harm; potential regulatory notification; containment within 4 hours
• P3 (Medium): Limited harm; no immediate regulatory obligation; investigation within 24 hours
• P4 (Low): Minor anomaly; no harm; investigation within 72 hours

AI-Specific Resolution Actions. The playbook library defines resolution actions specific to AI incidents:

• Model Rollback: Revert to prior approved model version (requires deployment token from GOV003)
• Scope Restriction: Limit model to lower-risk use cases while investigation proceeds
• Monitoring Enhancement: Increase monitoring frequency, log sampling, and alert sensitivity
• Emergency Human Override: Route all AI decisions to human review while model is investigated
• Input Filter: Deploy emergency input filter via GOV004 policy bundle update
• Third-Party Notification: Notify upstream model provider (for third-party model incidents)

5. Architecture Diagram

6. Components

7. Data Flow

P1 AI Incident Response Flow

8. Security Considerations

Incident Responder Access

• P1/P2 incidents require On-Call AI Incident Responder (24/7 coverage) with elevated read access to audit trail
• Elevated access is time-limited (incident duration + 48 hours); automatically revoked
• All actions during incident response logged with incident reference

OWASP LLM Top 10 Mapping

9. Governance Considerations

MTTD/MTTR Targets

Governance Artefacts

10. Operational Considerations

SLOs

11. Cost Considerations

Indicative Cost Range

12. Trade-Off Analysis

Option Comparison

13. Failure Modes

14. Regulatory Considerations

APRA CPS230

• §43: Operational incidents with material impact must be notified to APRA within 72 hours. AI incidents causing disruption to critical operations are material incidents.
• §46: Post-incident reviews required for material incidents. PIR template implements this requirement.

APRA CPS234

• §36: Cybersecurity incidents must be notified to APRA within 72 hours. AI security incidents (prompt injection attacks, model theft) are CPS234 notifiable.
• §37: Summary of cybersecurity incidents must be submitted to APRA annually.

EU AI Act

• Article 72: Providers of high-risk AI systems must report serious incidents (death/serious harm attributable to AI, fundamental rights infringements, significant property damage) to market surveillance authorities without undue delay and no later than 15 days after becoming aware.

Privacy Act 1988 / Notifiable Data Breaches (NDB)

• §26WK: Privacy disclosure via AI (memorisation, membership inference) triggers NDB assessment. 30-day notification obligation if likely to cause serious harm. Incident management workflow includes NDB assessment as mandatory step for privacy disclosure incidents.

15. Reference Implementations

AWS

Azure

On-Premises

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Proven (Level 3)

18. Revision History