Lambda Architecture
Run a fast approximate path and a slow accurate path side by side.
AdvancedData
A speed layer produces low-latency, approximate results from the live stream while a batch layer periodically recomputes exact results over the full dataset; a serving layer merges both for queries.
When to use it
- Consumers need low-latency results now, but also a periodically corrected, fully accurate view
- Batch reprocessing is affordable and fixes any drift the speed layer accumulates
Trade-offs
- Two codepaths (streaming and batch) implementing related logic, which can drift apart
- Operationally heavier than a single unified pipeline (see Kappa for the alternative)
Components used
Event StreamStream ProcessingData LakeBatch / Scheduled JobDocument Database
How it works
- A speed layer processes the incoming stream immediately, producing approximate, low-latency results.
- A batch layer independently reprocesses the complete historical dataset on a schedule, producing accurate results that overwrite the approximations.
- A serving layer merges both, giving queries recent-but-approximate data at the head and correct data for everything older.
Used in the wild
- Analytics where a rough live number is needed now and an exact number is needed for billing later.
- Systems whose stream processing cannot guarantee exactly-once, so batch acts as the correcting authority.
- Legacy migrations where a mature batch pipeline exists and streaming is being added alongside it.
Good to know
- Its defining flaw is that business logic is implemented twice, in two different engines, and the two implementations drift.
- Jay Kreps proposed Kappa architecture in 2014 as the rebuttal: keep only the stream, and reprocess by replaying it from the start. Most greenfield systems now go that way.
Related patterns
Retrieval-Augmented Generation (RAG)
Ground an LLM's answers in retrieved, up-to-date, private documents.
Vector Search + Rerank
Cheaply retrieve a broad candidate set, then precisely re-rank the top results.
Feature Store
Compute features once, serve them consistently to training and inference.
CQRS
Separate models and stores for writes and reads.