[EAAPL-DAT008] Real-Time Feature Engineering

Category: Data Architecture
Sub-category: Feature Engineering / Online ML Serving
Version: 1.3
Maturity: Proven
Tags: feature-engineering, feature-store, real-time, Kafka, Flink, Redis, point-in-time-correctness, feature-drift
Regulatory Relevance: EU AI Act Article 10, APRA CPS 234 (operational resilience), ISO 42001 §8.4

1. Executive Summary

AI models that make real-time decisions — fraud detection, personalised recommendations, credit pre-screening — require features computed from the most current available data. Batch feature engineering, where features are pre-computed overnight, introduces staleness that degrades model performance in time-sensitive use cases: fraud features computed 8 hours ago miss the most recent spending patterns that indicate fraud.

This pattern defines a production real-time feature engineering architecture that delivers features within milliseconds to inference endpoints, with freshness SLAs as tight as 10 seconds from event to serving. It covers the streaming pipeline (Kafka/Flink for feature computation), the dual-layer feature store (Redis for online serving, object store for offline training), point-in-time correctness for training data consistency, feature freshness SLA management, and feature drift detection.

Organisations that implement this pattern achieve fraud detection precision improvements of 15–25% through fresher features, and reduce time-to-feature from hours to seconds for real-time AI use cases.

Target audience: ML Platform leads, Data Engineering leads, Enterprise Architects.

2. Problem Statement

Business Problem

Real-time AI decisions are only as good as the data behind them. Stale features cause models to miss current context: a customer who spent $5,000 in the last 10 minutes looks identical to a normal customer on yesterday's features. Business consequences include missed fraud, poor real-time personalisation, and incorrect risk assessments.

Technical Problem

• Batch feature pipelines (nightly ETL) produce features that are 8–24 hours stale by inference time.
• Ad hoc feature computation at inference time introduces latency (50–500ms per lookup) and creates unmanaged technical debt.
• Features computed differently for training (batch) vs. serving (real-time) introduce training-serving skew — one of the top causes of production model underperformance.
• Point-in-time correctness is violated: training data uses features computed at the current time, not the time of the historical event, leaking future information into training.
• Feature pipelines are duplicated across teams; no reuse; inconsistent feature definitions.

Symptoms

• Fraud detection model AUC degrades during peak transaction periods (features are stale).
• Model performs well in offline evaluation but poorly in production (training-serving skew).
• Multiple teams computing the same features with slightly different logic.
• Feature pipeline failures cause model fallback to default values with no alerting.
• Training labels point-in-time valid but features are current-time (future data leakage detected in post-hoc audit).

Cost of Inaction

3. Context

When to Apply

• AI models making real-time decisions where feature freshness affects quality (fraud, recommendations, pricing, content ranking).
• Models where latency budget for inference is <100ms total.
• Organisations with >3 ML teams benefiting from shared feature definitions.
• Use cases requiring training-serving consistency (training-serving skew is a documented issue).

When NOT to Apply

• Batch inference where features can be pre-computed (nightly batch is sufficient).
• Very simple AI with 1–3 features that can be trivially computed at request time.
• Organisation at early AI maturity where a shared feature store adds operational complexity beyond team capability.

Prerequisites

Industry Applicability

4. Architecture Overview

Design Philosophy

The real-time feature engineering architecture solves the fundamental tension between feature freshness and serving reliability by maintaining two feature stores simultaneously — an online store for low-latency serving and an offline store for training data — with a shared computation layer ensuring they produce identical feature values.

Stream-First, Batch-Fallback Design. Features are computed from streaming events (Kafka) using Flink streaming jobs. This provides freshness from seconds (aggregated windows) to sub-second (lookup features). For features that cannot be computed from streams (complex aggregations requiring full historical data), a batch top-up runs at configurable intervals (hourly, daily) to pre-compute the batch component, which is then merged with the streaming-computed component in the feature store.

Dual-Layer Feature Store. The architecture maintains two feature store layers with different characteristics:

• Online store (Redis): Serves features at inference time with <10ms p99 latency. Stores only the most recent feature values per entity (customer_id, account_id, etc.). Redis sorted sets serve time-windowed aggregations. Optimised for point-in-time latest reads.
• Offline store (object store + columnar format): Stores the full history of feature values with timestamps. Used for training dataset creation. Enables point-in-time correct feature retrieval: given a historical event at time T, retrieve the feature value that would have been available at time T — not the current value.

Training-Serving Consistency via Shared Computation Logic. The most critical architectural property: the Flink streaming job and the offline feature pipeline must use identical computation logic. This is achieved through a shared Feature Transformation Library — a versioned Python/Java library containing the feature computation logic, consumed by both the Flink streaming job and the batch offline pipeline. When feature logic changes, both pipelines are updated together, eliminating training-serving skew by construction.

Point-in-Time Correctness. When creating a training dataset, features must be retrieved at the time of the label event, not the current time. The offline store's timestamp index enables point-in-time queries: for a fraud label at 2024-03-15 14:32:00, retrieve the feature values that existed at that exact time. This requires the offline store to retain historical feature snapshots (not just the latest value) — a storage cost trade-off managed through tiered retention (recent history at full fidelity; older history at reduced fidelity or summarised).

Feature Freshness SLA Management. Each feature has a declared freshness SLA — the maximum acceptable age of a feature value when served at inference time. The feature store monitoring layer continuously tracks feature ages and alerts when staleness exceeds SLA. Inference services receive feature freshness metadata alongside feature values, enabling them to downgrade confidence or trigger fallback behaviour when features are stale beyond SLA.

Feature Drift Detection. Feature values are monitored for distribution drift using Population Stability Index (PSI) computed over rolling windows. A feature whose distribution shifts significantly (PSI > 0.1) may indicate upstream data pipeline issues, concept drift, or business context changes. Drift alerts are routed to the feature owner for investigation.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Flow

Error Flow

8. Security Considerations

Authentication & Authorisation

• Redis online store access requires authentication; separate read-only credentials for inference services vs. read-write for feature pipelines.
• Feature Registry access control: feature ownership declaration restricts writes; reads are broadly accessible within organisation.

Secrets Management

• Redis credentials, Kafka credentials stored in secrets manager; rotated quarterly.

Data Classification

• Online feature store contains derived features from personal data; classified as Confidential minimum.
• Offline feature store contains historical personal data features; strict access control; encryption.

Encryption

• Redis at rest encrypted (Redis Enterprise AES-256); TLS for Redis connections.
• Offline feature store (S3/GCS) encrypted at rest; in transit TLS 1.3.

Auditability

• Feature access at inference time logged per entity ID (for right-to-explanation requests).
• Feature schema changes logged in Feature Registry with version history.

OWASP LLM Top 10 Mapping

9. Governance Considerations

Responsible AI

• Feature definitions must document what personal data they are derived from; subject to privacy review.
• Features based on protected attributes (age, ethnicity, gender, postcode as proxy) require explicit bias review before use in consequential AI.

Model Risk Management

• Feature freshness SLA is a model risk parameter: breach triggers model risk review.
• Training-serving skew detection is a model risk control; regular skew audits (quarterly) required.

Human Approval Checkpoints

• New feature requiring personal data processing: Privacy Officer review.
• Feature based on protected attribute proxy: Bias Review Board sign-off.
• Freshness SLA reduction (making SLA tighter): capacity review by ML Platform.

Governance Artefacts

10. Operational Considerations

Monitoring

SLOs

Capacity Planning

• Redis: size based on peak entity count × feature count × average feature value size × replication factor.
• Flink: size based on peak event throughput × window size complexity.
• Offline store: size based on entity count × feature count × retention period × snapshot frequency.

Disaster Recovery

11. Cost Considerations

Cost Drivers

Optimisations

• Cache frequently accessed entity feature vectors at inference service level (TTL = freshness SLA) to reduce Redis load.
• Use Redis Cluster hash slots to partition entities across nodes; avoids hot-key issues for high-velocity entities.
• Compress feature values in Redis using MessagePack or CBOR; typically 3–5× storage reduction.
• Tier offline store: recent 90 days at full fidelity; older at reduced granularity.

Indicative Cost Range

12. Trade-Off Analysis

Option Comparison

Architectural Tensions

13. Failure Modes

14. Regulatory Considerations

15. Reference Implementations

AWS

Azure

GCP

On-Premises

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Proven — Real-time feature engineering is production-proven at scale (Netflix, Uber, LinkedIn). Managed feature stores (Vertex AI, SageMaker) are GA. Operational complexity remains high for self-managed deployments.

18. Revision History