EAAPL-MDL001 — Model Versioning and Lineage Tracking

1. Executive Summary

AI models are long-lived, continuously evolving artefacts. Without systematic versioning, organisations cannot answer the two questions regulators and boards ask first: which model made that decision? and who approved it? EAAPL-MDL001 establishes a disciplined versioning schema adapted from semantic versioning principles, a co-located artefact bundle (weights, configuration, tokeniser, preprocessing code, evaluation results), and an approval workflow that gates every version before it reaches production. By registering every version in the enterprise Model Register (EAAPL-GOV001), the organisation achieves full lineage — from training data provenance through evaluation evidence to production retirement. The pattern is deliberately low-complexity: it imposes process rigour, not infrastructure cost. CIOs gain a defensible audit trail for internal audit and external regulatory inquiries. CTOs gain a reproducible deployment pipeline and the ability to perform rapid rollback. Risk officers gain the model card evidence needed for model risk management frameworks. Any organisation deploying more than one AI model to production should treat this pattern as a foundation — it is a prerequisite for shadow deployment (EAAPL-MDL002), canary release (EAAPL-MDL003), and rollback (EAAPL-MDL004).

2. Problem Statement

2.1 Business Problem

Product and risk teams cannot determine which model version is serving a given business function. When a customer complaint or regulatory inquiry arrives, the response is "we think it was the model from Q3" — an answer that destroys regulator confidence and exposes the organisation to material censure.

2.2 Technical Problem

Model artefacts (weights, configuration, evaluation results) are stored in ad-hoc locations — cloud storage buckets without naming conventions, local developer machines, CI/CD pipeline caches. Versions are identified by timestamps or commit hashes that carry no semantic meaning. There is no link between a deployed model and the training run, dataset, or evaluation that produced it.

2.3 Symptoms

• Incident post-mortems cannot identify which model version caused the issue.
• Re-running training with "the same" code and data produces a different model with no explanation.
• Multiple teams deploy models with the same name but different capabilities, causing downstream consumer confusion.
• Evaluation results exist in spreadsheets disconnected from the deployed artefact.
• Deprecation is ad-hoc: consumers discover a model has been retired when their API calls start failing.

2.4 Cost of Inaction

3. Context

3.1 When to Apply

• Any organisation deploying one or more AI/ML models to production systems.
• Teams preparing for regulatory examination under EU AI Act, APRA CPS234, or equivalent.
• Organisations implementing a Model Risk Management framework.
• Prior to implementing shadow deployment, canary release, or ensemble patterns.

3.2 When NOT to Apply

• Pure research/experimentation environments with no production deployment path (apply lightweight notebook versioning instead).
• Models trained and discarded within a single batch pipeline with no reuse intent.

3.3 Prerequisites

3.4 Industry Applicability

4. Architecture Overview

4.1 Versioning Schema

Model versioning in an enterprise context adapts semantic versioning (SemVer) to the unique characteristics of machine learning models. A version takes the form MAJOR.MINOR.PATCH[-suffix].

MAJOR version increment signals an architecture change that breaks backward compatibility for consumers. Examples: replacing a transformer encoder with a decoder-only architecture; switching from a BERT-class model to a GPT-class model; changing the output schema (e.g., adding a confidence distribution to a previously point-estimate model). Consumers must update their integration when MAJOR changes.

MINOR version increment signals a significant capability change that is backward-compatible at the API level. Examples: fine-tuning on a new domain dataset that meaningfully shifts output distribution; retraining with 12 additional months of data; updating the base model (e.g., from GPT-4o to GPT-4.1) while preserving the same prompt interface. Consumers should validate that downstream quality expectations are still met.

PATCH version increment signals a performance or efficiency change that does not alter model behaviour materially. Examples: INT8 quantisation of weights; ONNX export optimisation; vocabulary trimming; prompt template correction for a minor wording issue. Consumers are expected to accept patch upgrades without re-validation.

A suffix carries pre-release or environment signals: -alpha, -beta, -rc.1, -shadow (currently in shadow testing per EAAPL-MDL002), -deprecated.

4.2 Artefact Bundle

Every model version consists of a versioned artefact bundle — a single, immutable unit of content that travels together through the pipeline. The bundle contains:

• Weights: serialised model weights in a standard format (SafeTensors preferred; ONNX for cross-framework deployments).
• Configuration: model architecture config, hyperparameters, tokeniser vocabulary, and special tokens map.
• Preprocessing code: the exact pre/post-processing code used during training and inference, pinned by commit hash.
• Evaluation results: structured JSON/YAML file containing all evaluation metric values, benchmark dataset references, and evaluation date.
• Model card: structured Markdown document (see Section 4.3) authored and reviewed for this version.
• Provenance manifest: training run ID, training data references (dataset name + version + hash), compute environment (hardware type, framework versions, OS).

The bundle is stored as an immutable archive in object storage with a content-addressable hash. The hash is registered in EAAPL-GOV001 — any tampering is detectable.

4.3 Model Card Standards

Each version requires a model card authored by the team and reviewed by the AI governance function. The model card captures: intended use (primary use case, out-of-scope uses explicitly listed); training data description (provenance, size, temporal range, known biases); evaluation results (quantitative metrics on held-out test set, fairness metrics by demographic subgroup where applicable); known limitations; ethical considerations and risk rating; approval status and approver identity.

4.4 Approval Workflow

No model version may be promoted to a production environment without approval. The approval workflow is triggered by CI/CD on successful evaluation. Low-risk models require sign-off from the model owner and a peer reviewer. High-risk or regulated models additionally require sign-off from the AI Governance function and, for APRA-regulated entities, the Chief Risk Officer or delegate. Approval is recorded in EAAPL-GOV001 with timestamp and approver identity.

4.5 Deprecation Management

Deprecation follows a structured lifecycle: (1) Deprecation notice issued to all registered consumers at least 90 days before sunset for production models, 30 days for non-production. (2) Traffic migration plan agreed with consumers, including a canary ramp schedule per EAAPL-MDL003. (3) Sunset date recorded in the model register. (4) On sunset, the model is removed from serving infrastructure but the artefact bundle remains in cold storage for the regulatory retention period (minimum 7 years for regulated decisions).

5. Architecture Diagram

6. Components

7. Data Flow

7.1 Primary Flow

7.2 Error Flow

8. Security Considerations

8.1 Controls Summary

8.2 OWASP LLM Top 10 Relevance

9. Governance Considerations

9.1 Responsible AI

Every model version requires a completed model card that addresses: fairness evaluation results (disaggregated by relevant subgroups), intended and prohibited use cases, known biases, and human oversight requirements. The AI Governance function reviews model cards for high-risk models before approval.

9.2 Model Risk Management

Model versioning is the foundation of the organisation's Model Risk Management framework. Each version is a distinct model for MRM purposes. Version MAJOR increments trigger full model validation. MINOR increments trigger targeted validation against changed capability. PATCH increments require regression testing evidence only.

9.3 Human Approval Gates

No automated process may promote a model to production without a recorded human approval. The approval record must include: approver identity (not a service account), approval date, version being approved, risk tier assessment, and any conditions or exceptions noted.

9.4 Governance Artefacts

10. Operational Considerations

10.1 SLOs

10.2 Monitoring and Logging

The Model Register emits structured events on: version registration, approval state change, deployment, deprecation notice, and sunset. Events are forwarded to the organisation's SIEM. Alerts are configured for: unapproved versions deployed to production (P1), artefact hash mismatch (P1), overdue approval (P2).

10.3 Incident Response

A model-version incident (wrong version in production, unapproved version detected) triggers the AI Incident Response playbook. The immediate action is traffic shift to the last known-good approved version per EAAPL-MDL004. Root cause analysis is mandatory and must be completed within 5 business days.

10.4 Disaster Recovery

10.5 Capacity Planning

Model artefact bundles range from 500 MB (small fine-tuned models) to 150 GB (large foundation model fine-tunes). Storage growth is predictable: estimate 3–5 new versions per model per quarter. Retention policy: active versions indefinitely; deprecated versions 7 years cold storage. Plan for 2–5 TB/year of new artefact storage for a 10-model portfolio.

11. Cost Considerations

11.1 Cost Drivers

11.2 Scaling Risks

Evaluation compute is the primary scaling risk. As model size grows, full benchmark evaluation can take hours and require expensive GPU instances. Organisations must plan evaluation infrastructure as part of training budget, not as an afterthought.

11.3 Optimisations

• Use spot/preemptible instances for evaluation runs (evaluation is restartable).
• Tier evaluation depth by risk: PATCH versions run a reduced evaluation suite (subset of benchmarks); MAJOR versions run the full suite including adversarial tests.
• Implement incremental evaluation: cache results for unchanged benchmark subsets.
• Cold-tier deprecated artefacts immediately on sunset (typically 80% cost reduction vs hot storage).

11.4 Indicative Cost Range

12. Trade-Off Analysis

12.1 Versioning Schema Options

12.2 Architectural Tensions

13. Failure Modes

13.1 Cascading Failure Scenarios

If the Model Register becomes unavailable, deployment pipelines lose the ability to verify approved status of model versions. Without a circuit breaker, pipelines may fall back to deploying unverified versions. Mitigation: deployment pipelines cache the last-known approved version manifest locally with a 24-hour TTL; any deploy during register outage uses the cached manifest and triggers a P2 incident for human review.

14. Regulatory Considerations

15. Reference Implementations

15.1 AWS

• Artefact Store: S3 with Object Lock (COMPLIANCE mode) + S3 Intelligent-Tiering for cost.
• Model Register: Amazon SageMaker Model Registry (native versioning + approval workflow).
• Evaluation: SageMaker Processing Jobs for benchmark runs on managed GPU instances.
• Approval Workflow: SageMaker Model Registry approval + SNS notifications to approvers.
• Audit Log: CloudTrail + S3 (immutable log bucket).

15.2 Azure

• Artefact Store: Azure Blob Storage with Immutability Policies + Lifecycle Management.
• Model Register: Azure Machine Learning Model Registry.
• Evaluation: Azure ML Pipelines on GPU compute clusters.
• Approval Workflow: Azure ML + Power Automate for approval routing.
• Audit Log: Azure Monitor + Log Analytics Workspace.

15.3 GCP

• Artefact Store: Cloud Storage with Object Retention + Nearline/Coldline tiers.
• Model Register: Vertex AI Model Registry.
• Evaluation: Vertex AI Pipelines on A2/T4 instances.
• Approval Workflow: Vertex AI Model Registry approval states + Cloud Functions for routing.
• Audit Log: Cloud Audit Logs + BigQuery export.

15.4 On-Premises / Hybrid

• Artefact Store: MinIO (S3-compatible) with WORM buckets or NetApp StorageGRID.
• Model Register: MLflow Tracking Server (self-hosted, PostgreSQL backend).
• Evaluation: Kubernetes Jobs on GPU node pools.
• Approval Workflow: Jira Service Management or custom workflow microservice.
• Audit Log: Elasticsearch + Kibana or Splunk.

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Mature

18. Revision History