[EAAPL-DAT003] Data Lineage for AI

Category: Data Architecture
Sub-category: Data Lineage / AI Traceability
Version: 1.2
Maturity: Proven
Tags: data-lineage, provenance, OpenLineage, explainability, impact-analysis, traceability, regulatory-audit
Regulatory Relevance: EU AI Act Articles 12 & 13, APRA CPS 234, GDPR Article 22, ISO 42001 §8.6, NIST AI RMF GOVERN-6

1. Executive Summary

Regulators, auditors, and risk committees increasingly demand that organisations explain not just what an AI model predicted, but why — tracing back through the model architecture, training data, and ultimately to raw source systems. Traditional data lineage tools capture ETL pipelines but stop at the data warehouse boundary, leaving the AI layer invisible.

This pattern defines an end-to-end AI data lineage architecture using the OpenLineage standard, capturing lineage from raw source systems through every transformation stage, training run, model version, and inference event. The lineage graph enables regulators to answer: "Which version of which data, processed how, produced this model, which made this prediction on this date?"

Beyond compliance, AI lineage delivers operational value: when a source system changes, impact analysis identifies all downstream models at risk within minutes instead of weeks. Organisations adopting this pattern have reduced regulatory investigation response time from weeks to hours and eliminated surprise model breakages from upstream schema changes.

Target audience: Chief Data Officers, Chief Compliance Officers, Enterprise Architects, ML Platform leads.

2. Problem Statement

Business Problem

When an AI model makes a consequential decision (loan rejection, insurance claim denial, clinical risk assessment), the organisation must be able to explain and defend that decision — including the data that trained the model. Without lineage, this explanation is impossible, creating regulatory and legal exposure.

Technical Problem

• Data lineage tools capture ETL/SQL lineage but do not model AI-specific lineage events (feature engineering, training, inference).
• Model registries store model artefacts but do not link models to the specific dataset versions used for training.
• Inference logs record predictions but not the feature values used to produce each prediction, nor the model version.
• Impact analysis of upstream data changes on downstream AI models is manual and error-prone.
• No standard schema exists for capturing AI lineage events — leading to bespoke, non-interoperable solutions.

Symptoms

• Regulatory enquiry response ("explain this credit decision") takes weeks rather than hours.
• Schema change in an operational database silently breaks a downstream AI model.
• Model retraining produces different results but the cause cannot be traced to a specific data change.
• Multiple incompatible lineage stores exist: one for ETL, one for ML pipelines, one for BI — no unified view.
• Audit finds model was trained on data that included consent-withdrawn records; no mechanism to detect this.

Cost of Inaction

3. Context

When to Apply

• Any production AI system in a regulated industry.
• AI systems where decisions are consequential (credit, insurance, clinical, employment).
• Organisations with multiple AI models consuming data from shared source systems (high impact analysis value).
• Systems subject to right-to-explanation requirements (GDPR, EU AI Act, Privacy Act).
• Organisations where data contracts / data mesh are in use (lineage complements contract governance).

When NOT to Apply

• Pure research/PoC AI with no production decisions.
• AI systems consuming only fully external, black-box data APIs with no lineage available.
• Very simple AI systems (single feature, deterministic rule) where lineage is obvious from inspection.

Prerequisites

Industry Applicability

4. Architecture Overview

Design Philosophy

The foundational insight of this pattern is that AI lineage is a graph, not a table. The lineage of a single prediction involves a directed acyclic graph connecting: raw source records → data transformations → joined datasets → feature engineering → training dataset version → model training run → model version → inference request → prediction output. Each node in this graph is an immutable versioned artefact; each edge is a transformation event with metadata.

OpenLineage as the Standard. Rather than inventing a proprietary lineage schema, this pattern adopts the OpenLineage standard (openlineage.io), which defines a common event schema for lineage capture across data pipelines, ML platforms, and inference services. OpenLineage events are emitted by each pipeline stage, collected by a lineage backend (Marquez or Atlan), and stored in a queryable lineage graph.

Four Lineage Event Classes. AI lineage comprises four distinct event classes, each requiring specific schema extensions to the base OpenLineage spec:

1. Dataset lineage events: Emitted by ETL/ELT pipelines; capture source → transformation → output dataset with row counts, schema versions, and quality check results.
2. Feature engineering events: Emitted by feature pipelines; capture source datasets → feature computation logic → feature set version with temporal validity metadata.
3. Training events: Emitted by training pipelines; capture feature set versions → training run parameters → model artefact version with quality scorecard ID.
4. Inference events: Emitted by inference services; capture model version + feature values → prediction output with confidence score. Note: inference events are high-volume; sampling strategies are required for cost management while preserving full lineage for flagged or high-stakes predictions.

Lineage Query Patterns. Three primary query patterns drive the lineage architecture's design:

• Forward impact query: "Which models and predictions are downstream of dataset X?" — used for impact analysis before schema changes.
• Backward provenance query: "What data produced this prediction?" — used for regulatory explanation.
• Cross-version diff query: "What changed in the data between model version M1 and M2?" — used for model quality investigation.

These query patterns require a graph-capable storage backend (Neo4j, Amazon Neptune, or a columnar store with graph query extensions).

Selective Inference Lineage. Full capture of feature values at inference time for every prediction is prohibitively expensive at scale (millions of predictions/day). The pattern uses a tiered strategy: full lineage for consequential predictions (flagged by risk score, decision type, or regulatory classification); sampled lineage (1–5%) for routine predictions; and always-full lineage for predictions that are later reviewed, appealed, or investigated.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Flow

Error Flow

8. Security Considerations

Authentication & Authorisation

• OpenLineage Collector API requires authenticated emitters (API keys per pipeline stage); keys rotated quarterly.
• Lineage Query API requires role-based access: data engineers (full read), auditors (read-only subset), business analysts (anonymised lineage).
• Feature value snapshots in inference lineage classified as Confidential; access restricted to authorised investigators.

Secrets Management

• Collector API keys stored in secrets manager; not in pipeline code.
• Lineage graph database credentials rotated every 90 days.

Data Classification

• Lineage metadata (dataset names, row counts, schema versions) classified as Internal.
• Feature value snapshots in inference lineage classified as Confidential (may contain PII); stored with encryption and strict access control.
• Regulatory reports generated from lineage may contain PII context; classified as Confidential.

Encryption

• Lineage graph store encrypted at rest (AES-256); in transit TLS 1.3.
• Feature value snapshots encrypted at rest with separate encryption keys; key access logged.

Auditability

• All lineage query executions logged (who queried what lineage, when, why).
• Lineage events immutable once written; no update/delete path.
• Lineage completeness gaps (missing events) logged and alerted.

OWASP LLM Top 10 Mapping

9. Governance Considerations

Responsible AI

• Complete lineage enables bias attribution: if a model exhibits demographic bias, lineage identifies whether the bias originates in source data, feature engineering, or labelling.
• Lineage enables right-to-erasure impact analysis: when a data subject requests erasure, lineage identifies all models trained on data linked to that subject (machine unlearning trigger).

Model Risk Management

• Model risk committees require provenance validation for all production models; Lineage Query API provides this programmatically.
• Model version deprecation requires impact analysis (which predictions were served by this version?); lineage graph enables this query.

Human Approval Checkpoints

• Before upstream schema change: Impact Analysis Service report reviewed by data owner + affected ML leads.
• Regulatory investigation: Lineage Query API output reviewed by compliance officer before submission to regulator.

Governance Artefacts

10. Operational Considerations

Monitoring

SLOs

Logging

• All OpenLineage events logged in raw form (JSON) alongside graph store; serves as event replay source.
• Lineage query audit log retained 7 years.

Incident Management

• Lineage gap (missing events for production model) → P2 incident; ML Platform investigates emitter health.
• Lineage graph store unavailable → P1 if regulatory investigation in progress; P2 otherwise.

Disaster Recovery

11. Cost Considerations

Cost Drivers

Scaling Risks

• Inference lineage feature snapshots at high prediction volume (>1M/day) can generate significant storage cost; use sampling + tiered retention.
• Graph store query cost grows with graph depth; optimise with indexed traversal patterns.

Optimisations

• Use OpenLineage open-source stack (Marquez + PostgreSQL backend) for cost-sensitive deployments.
• Apply sampling for routine inference lineage; full capture only for consequential predictions.
• Implement lineage graph compaction: archive lineage for deprecated model versions to cold storage.
• Cache frequent backward provenance queries for commonly audited decisions.

Indicative Cost Range

12. Trade-Off Analysis

Option Comparison

Architectural Tensions

13. Failure Modes

Cascading Failure Scenarios

• Schema change cascade without lineage: Data team changes source table schema → no impact analysis (lineage incomplete) → feature pipeline silently produces incorrect features → model degradation → consequential decisions affected → regulatory enquiry → unable to trace root cause.
• Lineage collector outage during audit: Regulator requests provenance report → Lineage Query API unavailable → raw event store must be replayed → hours of investigation delay.

14. Regulatory Considerations

15. Reference Implementations

AWS

Azure

GCP

On-Premises

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Proven — OpenLineage standard is mature and broadly adopted. Graph-based lineage storage is operationally proven. Inference-level lineage capture at scale remains an evolving practice.

18. Revision History