EAAPL-KNW005: Knowledge Graph for Explainability

Pattern ID: EAAPL-KNW005 Status: Emerging Complexity: High Tags: knowledge-graph explainability eu-ai-act high-complexity Version: 1.0 Last Updated: 2026-06-12

1. Executive Summary

The Knowledge Graph for Explainability pattern uses a structured knowledge graph to make AI reasoning transparent, traceable, and comprehensible to both end users and regulators. When an AI system produces a recommendation or decision, this pattern generates a human-readable explanation by tracing the reasoning path through the knowledge graph: identifying which facts supported the conclusion, how those facts relate to each other, and what the chain of inference was from evidence to outcome.

This pattern is specifically designed to meet the explainability requirements of the EU AI Act (Article 13 transparency, Article 14 human oversight), GDPR Article 22 (automated decision explanation), and financial services model risk management regulations (SR 11-7, APRA). Unlike post-hoc explanation methods (SHAP, LIME) that approximate explanations for opaque models, knowledge graph reasoning is inherently traceable — explanations are derived from the actual reasoning path, not an approximation of it.

For CIOs and CTOs in regulated industries, this pattern closes the "explainability gap" that blocks high-risk AI use cases from production deployment. Medical triage, credit decisioning, insurance underwriting, and compliance monitoring are all domains where AI could deliver substantial value but regulatory requirements for explanation have blocked deployment. Knowledge graph reasoning makes explanation structural, not aspirational.

Implementation requires the Enterprise Knowledge Graph (EAAPL-KNW001) as a prerequisite, adding an explanation generation and presentation layer on top.

2. Problem Statement

2.1 Business Problem

High-value AI use cases in regulated industries — credit decisioning, medical diagnosis assistance, insurance underwriting, legal compliance monitoring — remain in pilot or PoC status not because the AI lacks accuracy but because organisations cannot explain AI conclusions to affected individuals, regulators, and oversight bodies. The inability to explain means the inability to defend, and the inability to defend means the inability to deploy.

2.2 Technical Problem

Large language models are fundamentally opaque: the internal reasoning process that produces a conclusion is not directly inspectable or extractable. Post-hoc explanation methods (SHAP, LIME, attention visualisation) approximate what inputs influenced the output but do not reconstruct the actual reasoning chain. For high-stakes decisions, an approximation of the reasoning is legally and practically insufficient — the explanation must accurately represent why the system reached the conclusion, not a statistical approximation of the contributing factors.

2.3 Symptoms

• High-risk AI applications stuck in pilot indefinitely because legal/compliance cannot accept opaque AI decisions
• AI decisions challenged by customers or regulators but the organisation cannot provide a satisfactory explanation
• Compliance team manually reviewing and re-justifying every AI recommendation before it is acted upon — eliminating the efficiency gain
• AI governance programme cannot document AI decision logic at the level required by audit

2.4 Cost of Inaction

• EU AI Act Article 13 non-compliance for high-risk AI systems carries fines of up to €30M or 6% of global annual turnover
• GDPR Article 22 violations for automated decision-making without meaningful explanation carry fines of up to €20M or 4% of annual global turnover
• Continued human-in-the-loop overhead for all high-risk AI decisions, eliminating the efficiency case for AI investment
• Competitor organisations that solve explainability first can deploy high-risk AI at scale and gain a structural advantage

3. Context

3.1 When to Apply

• AI systems making or recommending decisions that materially affect individuals (credit, insurance, medical, employment)
• Regulated AI use cases where regulators require or will require documented AI decision reasoning
• Domains where domain experts need to understand and validate AI reasoning before acting on recommendations
• Organisations already operating an Enterprise Knowledge Graph (EAAPL-KNW001) — this pattern extends it
• AI applications where user trust requires transparency, even in the absence of regulatory mandate

3.2 When NOT to Apply

• Low-risk AI use cases (content recommendations, search ranking) where informal explanation is sufficient
• Organisations without an Enterprise Knowledge Graph — the prerequisite investment is substantial; this pattern alone cannot justify it
• Real-time decisions with sub-100ms latency requirements — explanation generation adds latency incompatible with hard real-time constraints
• Creative or generative AI tasks (writing, image generation) where "reasoning" is not a well-defined concept

3.3 Prerequisites

• Enterprise Knowledge Graph (EAAPL-KNW001) operational in production with sufficient domain coverage
• AI application that produces decisions or recommendations based on knowledge graph traversal (not a pure neural network)
• Regulatory analysis confirming which explanation obligations apply (EU AI Act risk classification, GDPR Art. 22 automated decision scope)
• UX design for explanation presentation adapted to target audience (consumer, professional, regulator)

3.4 Industry Applicability

4. Architecture Overview

The Knowledge Graph for Explainability architecture adds four layers to the Enterprise Knowledge Graph: Reasoning Path Tracing, Evidence Chain Assembly, Natural Language Explanation Generation, and Explanation Presentation.

4.1 AI Application with Graph-Native Reasoning

This pattern requires that the AI application makes decisions based at least partially on explicit knowledge graph traversal, not solely on LLM token prediction. The decision engine queries the knowledge graph using a defined reasoning strategy: rule-based traversal (if entity A has relationship R1 to entity B, and entity B has property P above threshold T, then conclusion C applies) combined with evidence-weighted inference (multiple supporting facts increase conclusion confidence; contradictory facts decrease it).

The decision engine does not act as a black box — it records every graph query it executes, every node and edge it traverses, and the inference steps from evidence to conclusion. This execution trace is the raw material for explanation generation.

4.2 Reasoning Path Tracing

The reasoning path tracer captures the execution trace from the decision engine in a structured format. For each conclusion the AI system produces, the trace records:

• The initial query (what question or decision trigger was received)
• The reasoning strategy applied (which rule set or inference pattern was invoked)
• The graph traversal sequence: ordered list of (node_id, edge_type, node_id, confidence) tuples traversed
• The facts retrieved at each step: (subject, predicate, object, source_document, confidence_score, effective_date)
• The inference steps: how retrieved facts combine to support or contradict the candidate conclusion
• The conclusion confidence score: weighted combination of supporting evidence confidence

The trace is immutable and stored alongside the decision record. It cannot be retroactively modified.

4.3 Evidence Chain Assembly

The evidence chain assembler converts the raw reasoning trace into a human-logical structure: an ordered sequence of premises and inference steps that a domain expert could follow.

Premise identification: extract the key facts from the reasoning trace that materially influenced the conclusion. Filter out traversal steps that were explored but did not contribute to the final conclusion. Rank remaining premises by their contribution weight.

Contradiction identification: identify any facts in the knowledge graph that contradict the conclusion. An honest explanation must surface these — omitting contradictory evidence from an explanation given to a regulator or affected individual is a governance failure.

Confidence propagation: each premise carries a confidence score. The evidence chain carries an overall conclusion confidence that reflects the weakest link in the chain — a conclusion is only as confident as its least-certain supporting premise.

Source attribution: each premise is attributed to its source document, with document version, section reference, and effective date. The evidence chain is a fully cited argument, not an uncited assertion.

4.4 Natural Language Explanation Generation

The evidence chain (structured premises + inference steps + source attributions) is converted to plain English by an LLM with a tightly constrained prompt. The prompt:

• Provides the evidence chain as structured input (not free-form text)
• Specifies the target audience: consumer (plain English, no jargon), professional (domain terminology, concise), regulator (formal language, explicit clause citations)
• Constrains the LLM to report only what is in the evidence chain — it must not add inferences, context, or conclusions not present in the chain
• Requires explicit attribution: every claim in the explanation must be traceable to a specific premise in the chain
• Includes uncertainty communication: if the conclusion confidence is below a threshold, the explanation must communicate this to the reader

The generated explanation is validated against the evidence chain before delivery: every claim in the natural language text is mapped back to a premise in the structured chain. Claims without a corresponding premise are rejected and the explanation is regenerated. This validation step prevents the LLM from hallucinating explanatory content.

4.5 Explanation Presentation

Explanations are presented through multiple channels adapted to the recipient:

Consumer channel: plain English explanation with reading level calibrated to the target demographic. Structured as: "This decision was made because..." followed by 2–4 key reasons in bullet form, each with a "Based on: [source]" citation. A "Why we're not certain" section is included if confidence is below threshold.

Professional channel: more detailed explanation showing the full reasoning chain with intermediate inference steps. Domain terminology used where appropriate. Evidence confidence scores displayed. Contradictory evidence explicitly flagged.

Regulator/audit channel: full structured evidence chain in machine-readable format (JSON-LD), plus the natural language explanation, plus the complete reasoning trace, plus references to the specific ontology nodes and source documents. This is the audit-ready package.

Interactive exploration: for professional users, an interactive graph visualisation that allows them to navigate the reasoning path through the knowledge graph — clicking on each node shows the underlying document evidence.

5. Architecture Diagram

6. Components

7. Data Flow

7.1 Primary Data Flow — Decision to Explanation

7.2 Error Flow

8. Security Considerations

8.1 Authentication and Authorisation

Explanation endpoints are access-controlled: the same authorisation policies that govern access to the underlying decision apply to access to the explanation. A customer can access the explanation for their own decision; they cannot access explanations for other customers' decisions. Internal professional users are ABAC-controlled by role and domain.

8.2 Secrets Management

Knowledge graph credentials, LLM API keys, and explanation store credentials are stored in a secrets vault. The explanation generation LLM receives only the structured evidence chain — no raw PII or sensitive system data — minimising the sensitivity of the LLM API call.

8.3 Data Classification

Explanations may contain personal data (the explanation for a credit decision references the individual's financial data). Explanation records are classified at the same level as the underlying decision data and subject to the same access controls, retention policies, and right-to-erasure procedures.

8.4 Encryption

Explanation store: encrypted at rest (AES-256, CMK). In transit: TLS 1.3. The reasoning trace record is particularly sensitive (it documents the organisation's decision logic) — store with restricted access beyond the decision record and explanation.

8.5 Auditability

Every explanation is linked to the specific decision record, the specific knowledge graph version (snapshot ID), the reasoning trace, and the LLM model version used for NL generation. This full package constitutes the audit record for any AI decision. Retention matches the decision record retention period (minimum 7 years for financial services decisions).

8.6 OWASP LLM Top 10 Mapping

9. Governance Considerations

9.1 Responsible AI

Explainability is a fundamental responsible AI requirement. However, explanation quality can itself introduce bias: if the evidence chain is presented selectively (omitting contradictions, emphasising supporting evidence), the explanation can be misleading even when technically accurate. The contradiction surfacing requirement (§4.3) is a mandatory honest-explanation control. Explanations must represent the AI's actual reasoning, including uncertainty, not a post-hoc rationalisation of a desired outcome.

9.2 Model Risk Management

The explanation generation LLM is a secondary model (it explains the primary decision model's reasoning). It carries its own model risk: it could misrepresent the evidence chain through hallucination. The explanation validator (§4.4) is the primary model risk control for the explanation LLM — it verifies that the NL explanation is a faithful representation of the structured evidence chain. The validator's performance is tracked on a golden explanation set.

9.3 Human Approval Gates

For high-risk decision categories (credit refusal, insurance claim denial, medical triage), the explanation is reviewed by a qualified human professional before delivery to the affected individual. This review does not change the decision (the AI decision stands) but validates that the explanation is accurate, complete, and appropriate in tone. The reviewer can flag explanation quality issues for remediation without overriding the AI decision.

9.4 Policy Ownership

Explanation policy (what level of detail is required for each decision category and recipient type; what language is appropriate; what constitutes an adequate explanation for regulatory purposes) is owned by the Legal and Compliance function in collaboration with the AI Governance team. The AI Platform Engineering team owns the technical implementation. The business unit owns the decision process that generates the explanations.

9.5 Traceability

The complete audit package for any AI decision includes: the decision record, the reasoning trace, the evidence chain, the NL explanation (all versions generated), the validation results, and the knowledge graph snapshot ID at time of decision. This package is linked by a common decision_id and retrievable in its entirety by authorised investigators.

9.6 Governance Artefacts

10. Operational Considerations

10.1 Monitoring and SLOs

10.2 Logging

All explanation generation events are logged: decision_id, reasoning trace ID, evidence chain hash, NL explanation version, validation result, delivery channel, recipient role, timestamp. The explanation validator logs each claim-to-premise mapping result. All logs are immutable and retained per decision record retention schedule.

10.3 Incident Management

P1: AI decisions being made without traceable explanations (reasoning trace generation failure) — immediate halt of affected AI decision pipeline; human fallback. P2: Explanation validator failure rate above threshold — investigation of LLM model quality; temporary switch to structured-only explanations. P3: Consumer explanation language quality complaints — tone/reading level adjustment within policy guidelines.

10.4 Disaster Recovery

10.5 Capacity Planning

Explanation generation adds approximately 1–5 seconds of latency per decision depending on evidence chain complexity and LLM response time. At high decision volumes (>100/second), the explanation generation pipeline must scale horizontally. Explanation store grows linearly with decision volume: approximately 5–50 KB per decision explanation record (including trace and evidence chain). At 1 million decisions per month, expect 5–50 GB of explanation storage per month.

11. Cost Considerations

11.1 Cost Drivers

11.2 Scaling Risks

• At very high decision volumes (millions per day), LLM explanation generation cost becomes substantial — implement tiered explanation: automated NL generation for medium-risk decisions; structured-only for low-risk; full NL + human review only for high-risk
• Human review cost is the primary scaling constraint for high-risk decision categories — automation of explanation quality checking reduces but cannot eliminate this

11.3 Optimisations

• Cache explanations for structurally identical decision scenarios (same evidence chain, different individuals) — generate the explanation template once and fill individual-specific values
• Use shorter, cheaper models for initial explanation generation; route to more capable models only when validator rejects the first attempt
• Batch non-urgent explanation generation (e.g., explanations for decisions not yet requested by the individual) during off-peak hours at lower compute cost

11.4 Indicative Cost Ranges

12. Trade-Off Analysis

12.1 Explanation Approach Comparison

12.2 Architectural Tensions

13. Failure Modes

13.1 Cascading Failure Scenarios

Scenario 1: Knowledge Graph Staleness Cascade. A regulatory change alters the eligibility rules for a benefit. The knowledge graph is not updated. AI decisions continue using old rules. Explanations faithfully represent the old reasoning — they are internally consistent with the graph but externally incorrect relative to the new regulation. Users receive explanations citing outdated policy. Regulatory body conducts audit. Root cause: freshness management failure. Mitigation: regulatory corpus freshness SLA (see EAAPL-KNW003); regulatory change management process triggers knowledge graph update.

Scenario 2: Explanation Discrimination Discovery. A statistical audit of AI explanations reveals that the evidence chain systematically cites different factors for demographically different groups making equivalent requests. The knowledge graph contains historically biased relationship data that the AI inherited. The explanations are accurate (they truly reflect the reasoning) but the reasoning itself is biased. The transparency of the knowledge graph explanation system is what reveals the bias — which is a feature, not a bug. Resolution requires: knowledge graph bias audit; reweighting or removal of biased relationship data; decision impact assessment.

14. Regulatory Considerations

15. Reference Implementations

15.1 AWS

15.2 Azure

15.3 GCP

15.4 On-Premises

16. Related Patterns

17. Maturity Assessment

Overall Maturity Label: Emerging

18. Revision History