[EAAPL-GOV002] AI Risk Assessment Framework

Category: Governance / Risk Management Sub-category: Pre-deployment Risk Assessment Version: 2.0 Maturity: Mature Tags: risk-assessment, EU-AI-Act, NIST-AI-RMF, model-risk, fairness, safety, pre-deployment Regulatory Relevance: EU AI Act Articles 6–10, NIST AI RMF MAP Function, ISO/IEC 42001 §6.1.2, APRA CPS230 §19–§22

1. Executive Summary

The AI Risk Assessment Framework provides a systematic, repeatable methodology for evaluating AI system risk before any deployment into production environments. It addresses the fundamental challenge facing every regulated enterprise: how do you consistently determine whether an AI system is safe, fair, compliant, and fit-for-purpose—before customers and regulators are exposed to failures?

This pattern operationalises five risk dimensions—accuracy, fairness, security, compliance, and operational resilience—into a structured assessment workflow that produces a defensible, documented risk rating. The rating gates progression through the approval workflow (GOV003) and determines the intensity of post-deployment monitoring controls.

The framework directly implements the NIST AI RMF MAP function, satisfies EU AI Act Article 9 risk management system requirements for high-risk AI, and provides the risk classification evidence required by APRA CPS230. It is the second mandatory pattern in the EAAPL governance hierarchy, positioned immediately downstream of the AI Model Register (GOV001).

For the CIO/CTO audience, this pattern converts the abstract question "is our AI safe?" into a structured process with clear accountability, documented evidence, and regulatory defensibility. Organisations that implement this pattern reduce regulatory examination findings by eliminating the "we didn't formally assess risk" defence gap.

2. Problem Statement

Business Problem

AI systems are deployed based on technical performance metrics (accuracy, F1 score) without systematic assessment of business risk dimensions: fairness to customer segments, regulatory compliance, reputational exposure, and operational failure modes. When failures occur, there is no evidence that risk was considered before deployment.

Technical Problem

Risk dimensions for AI are heterogeneous and cannot be captured by a single metric. Accuracy metrics do not reveal discriminatory outcomes. Security assessments miss AI-specific threats (adversarial inputs, model inversion). Operational assessments miss AI-specific failure modes (distribution shift, hallucination). No single tool covers all dimensions; the framework must orchestrate multiple specialised assessment tools.

Symptoms

• AI systems deployed with only technical performance sign-off (accuracy, latency)
• Post-deployment discovery of fairness issues affecting customer segments
• Regulatory findings citing absence of pre-deployment risk documentation
• Security teams assessing AI systems using generic application security frameworks (missing AI-specific threat vectors)
• Inconsistent risk ratings across business units for functionally similar AI systems
• No traceability between deployment decision and risk evidence

Cost of Inaction

• Regulatory: EU AI Act Article 9 non-compliance for high-risk systems: fines up to €30M or 6% global turnover
• Legal: Discriminatory AI outcomes triggering class actions (financial services credit decisions, insurance pricing)
• Operational: Distribution shift causing silent model degradation undetected for months
• Reputational: Public disclosure of unfair AI outcomes; media scrutiny of automated decision-making

3. Context

When to Apply

• Before any AI system is promoted to a production or production-equivalent environment
• When an existing AI system undergoes material change (new training data, model architecture change, new use case, new user population)
• When a third-party AI system is onboarded for use in enterprise processes
• At periodic re-assessment intervals (annually for Low risk; 6-monthly for Medium; quarterly for High/Critical)
• When an AI incident triggers re-assessment per GOV008

When NOT to Apply

• Pure rules-based systems with no statistical learning component (standard risk assessment applies)
• Internal-only development and test environments with no customer data and no production traffic
• Trivial AI features with zero decision-making authority (e.g., autocomplete suggestions with human confirmation)

Prerequisites

• Model registered in AI Model Register (GOV001) — MRID required
• Model owner and business sponsor identified
• Training data lineage documented
• Initial use case description available
• Access to model for technical testing (or vendor-provided technical documentation for third-party models)

Industry Applicability

4. Architecture Overview

The AI Risk Assessment Framework is architected as an assessment pipeline—a structured sequence of risk dimension evaluations that produces a composite risk rating. The pipeline is intentionally modular: each dimension has its own assessment module with defined inputs, methods, thresholds, and outputs. This modularity enables dimension-specific tooling, parallel execution where dimensions are independent, and incremental framework enhancement without rebuilding from scratch.

Assessment Philosophy: Evidence-Based, Not Opinion-Based. Each risk dimension requires quantitative evidence—test results, metric outputs, statistical analyses—not just attestations from model owners. This is the critical difference between a governance checkbox process and a genuinely risk-reducing framework. The assessment workflow enforces evidence upload before any dimension can be marked complete.

Five-Dimension Risk Model. The framework assesses five orthogonal risk dimensions:

Accuracy Risk measures whether the model performs at the level required for its intended use case. This is not simply "what is the accuracy percentage"—it requires understanding acceptable error rates in context. A 95% accurate fraud detection model that misclassifies $50M in transactions annually has a very different risk profile than a 95% accurate movie recommendation system. Accuracy assessment includes baseline performance, performance under distribution shift, confidence calibration, and failure mode analysis.

Fairness Risk assesses whether model outputs disadvantage identifiable groups in ways that violate anti-discrimination obligations or organisational values. The framework tests for demographic parity, equalised odds, and individual fairness across protected attributes (race, gender, age, disability status). The threshold for acceptable disparity is use-case dependent: credit decisioning and insurance pricing face stricter fairness obligations than personalised recommendations. The Model Bias Detection pattern (GOV006) provides the continuous post-deployment counterpart to this pre-deployment assessment.

Security Risk evaluates AI-specific threats that standard application security assessment misses: adversarial input attacks (crafted inputs that cause misclassification), model inversion attacks (extracting training data from model outputs), membership inference attacks (determining if specific data was in the training set), prompt injection for generative AI, and supply chain risks from third-party model weights. The assessment uses both static analysis and adversarial testing.

Compliance Risk determines whether the model's operation complies with applicable law and regulation: privacy obligations (data minimisation, purpose limitation), sector-specific requirements (responsible lending, insurance pricing regulations), automated decision-making rights (Privacy Act, GDPR Article 22), and geographic restrictions. Compliance risk is the only dimension where a single finding can be an absolute blocker regardless of performance on other dimensions.

Operational Risk assesses resilience: What happens when the model fails? How quickly does performance degrade under distribution shift? What is the fallback when the model is unavailable? Does the model have appropriate monitoring coverage? Operational risk is particularly important for AI systems embedded in critical business processes where model unavailability would constitute an operational disruption.

Composite Risk Rating. The five dimension scores are aggregated using a worst-case-dominated algorithm: the composite rating equals the highest individual dimension rating. This is architecturally deliberate—a model that is perfectly accurate and fair but has Critical security risk should be rated Critical overall. Business units cannot average-down a critical dimension finding.

Use-Case Context Adjustment. Raw dimension scores are adjusted for use-case context: a credit decision AI requires tighter fairness thresholds than an internal document classifier. Context vectors (customer-facing, automated decision, financial consequence, vulnerable population involvement) amplify or reduce threshold sensitivity. This prevents the framework from applying uniform thresholds to fundamentally different risk contexts.

Mandatory vs. Conditional Assessments. Not every dimension requires the same assessment depth for every model. The framework uses a risk-screening questionnaire to determine which assessment modules apply at full depth vs. abbreviated form. A low-risk internal productivity tool may complete Accuracy and Operational assessments at abbreviated depth, while a customer-facing credit model requires full depth across all five dimensions plus external expert review.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Assessment Flow

Error Flow

8. Security Considerations

Authentication & Authorisation

• Assessors must have Assessment Contributor role (granted per-assessment, time-limited)
• Evidence bundles accessible only to assessors on active engagement + Compliance Reader role
• External experts access via time-limited, scoped portal credentials; no enterprise SSO

Secrets Management

• Model API keys used for adversarial testing stored in Vault with per-assessment lease
• Test dataset S3 paths generated as pre-signed URLs with 7-day expiry

Data Classification

• Test datasets carrying personal information: CONFIDENTIAL; processed in isolated assessment environment
• Assessment reports: RESTRICTED until model approved/rejected; then CONFIDENTIAL
• Fairness test results containing demographic analysis: RESTRICTED (demographic inference risk)

Encryption

• All evidence at rest: AES-256
• Evidence in transit: TLS 1.3; assessor portal enforces HTTPS only
• Sensitive training data used for assessment: encrypted assessment sandbox with no internet egress

OWASP LLM Top 10 Mapping

9. Governance Considerations

Responsible AI Alignment

The framework operationalises all five responsible AI principles: accuracy (performance), fairness (bias testing), security (adversarial), compliance (regulatory), and operational (accountability + human oversight). No principle is assessed by attestation alone.

Model Risk Alignment

The framework produces the primary input to model risk classification. Risk rating feeds directly into:

• Model validation requirements (independent validation required for High/Critical)
• Human oversight intensity (automated decisions require human review for Critical-rated models)
• Monitoring frequency and alert thresholds

Human Approval Workflow Integration

Assessment output directly triggers GOV003 approval workflow. Risk rating determines approval authority level:

• Low: AI Governance lead sign-off
• Medium: AI Governance + Risk Officer
• High: AI Governance + Risk Officer + CISO
• Critical: Full approval board including CRO and General Counsel

Policy Integration

Compliance module references current versions of: AI Use Policy, Data Classification Policy, Geographic Restriction Register, Prohibited Use Case Register. Policy updates trigger re-assessment queue for affected deployed models.

Governance Artefacts

10. Operational Considerations

Monitoring

SLOs

Disaster Recovery

11. Cost Considerations

Cost Drivers

Indicative Cost Range

12. Trade-Off Analysis

Option Comparison

Architectural Tensions

13. Failure Modes

14. Regulatory Considerations

APRA CPS230

• §19: Risk management requires identification, assessment, and mitigation of operational risks. AI systems must be assessed before deployment as part of the operational risk framework.
• §22: Scenario analysis requirements: accuracy failure, fairness failure, and security compromise scenarios must be documented and tested.

EU AI Act

• Article 6 & Annex III: High-risk AI classification determines mandatory conformity assessment. Framework outputs the classification evidence.
• Article 9: High-risk AI systems must have a risk management system with continuous iterative process for identification, analysis, and evaluation of risks. This framework implements Article 9 requirements.
• Article 10: Training, validation, and testing data requirements. Accuracy module includes data quality assessment satisfying Article 10(2).

NIST AI RMF

• MAP 1.1–1.6: Context establishment and risk identification. Framework screening questionnaire and use-case context adjustment implement MAP function.
• MAP 2.1–2.3: Risk categorisation. Five-dimension scoring produces NIST-aligned risk categorisation.
• MEASURE 1.1–2.6: Metrics and measurement approaches for all five dimensions are defined and implemented.

ISO/IEC 42001:2023

• §6.1.2: AI risk identification and assessment. Framework is the primary implementation of this clause.
• §8.4: Operational planning and control for AI risk assessment processes.

Privacy Act 1988 / APPs

• APP 3: Collection of personal information. Compliance module checks that model input data collection meets purpose limitation and minimisation requirements.
• APP 11: Security. Security module assessment satisfies APP 11 technical security obligations for AI systems processing personal information.

15. Reference Implementations

AWS

Azure

GCP

Open Source Stack (On-Premises)

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Mature (Level 4)

18. Revision History