sydepsystem design atlas

Feature Store

Compute features once, serve them consistently to training and inference.

AdvancedAI / MLData
Batch FeaturesAnalytics & DataStreaming FeaturesAnalytics & DataFeature StoreAI / MLTraining (offline)AI / MLInference (online)AI / ML

Batch and streaming pipelines compute features into a shared store with both an offline (training) and online (low-latency serving) view, eliminating the training/serving skew that comes from two teams recomputing features differently.

When to use it

  • Multiple models need the same engineered features, computed consistently
  • Training/serving skew has caused production model quality to diverge from offline evaluation

Trade-offs

  • The store itself becomes shared infrastructure with its own reliability requirements
  • Feature definitions need governance or duplicate, inconsistent versions proliferate

Components used

Ingestion / ETL PipelineStream ProcessingFeature StoreModel TrainingLLM / Model Endpoint

How it works

  • Feature transformations are defined once and materialised into two stores: an offline store holding full history for training, and an online store holding only current values for low-latency inference.
  • Both are fed from the same definition, which is the entire point — the training pipeline and the serving path cannot drift apart.
  • The offline store must be able to reconstruct a feature's value as of a past timestamp, so training rows only ever see data that existed at prediction time.

Used in the wild

  • Any organisation with several teams reusing the same features, such as customer lifetime value or 30-day activity counts.
  • Fraud and risk scoring where the same aggregate must be identical during training and at decision time.
  • Recommendation systems needing sub-10ms lookups of precomputed user and item features.

Good to know

  • The problem this solves is training/serving skew: a feature computed one way in a batch SQL job and another way in application code, producing a model that scores well offline and badly in production.
  • Point-in-time correctness is the hard part. Joining a feature's current value onto a historical label leaks the future into training and produces spectacular, entirely fake offline accuracy.