Change Data Capture
Stream a database's row-level changes without touching app code.
IntermediateDataMessaging
A CDC connector reads the database's commit log and publishes every insert, update and delete as an event, letting downstream systems stay in sync without the source service having to publish anything itself.
When to use it
- You need to react to data changes in a legacy system you cannot modify
- Multiple downstream systems need the same stream of changes
Trade-offs
- Couples consumers to the source's physical schema, not a stable contract
- Connector lag and replay semantics need monitoring like any other pipeline
Components used
Relational DatabaseIngestion / ETL PipelineEvent StreamSearch IndexData Warehouse
How it works
- A connector tails the database's replication log — the write-ahead log or binlog — and converts committed row changes into a stream of events.
- Because it reads the log rather than the tables, it captures every change including those made by scripts or other applications, with no polling and no application code changes.
- Downstream consumers project those events into search indexes, caches, warehouses or other services.
Used in the wild
- Keeping a search index or cache in sync with a database without dual writes.
- Streaming operational data into an analytics warehouse in near real time.
- Strangling a legacy system by mirroring its data outward while it remains the system of record.
Good to know
- CDC is the honest alternative to dual writes. Writing to the database and then publishing an event is two operations without a shared transaction, and it will eventually leave the two out of sync.
- Schema changes upstream will break downstream consumers. A schema registry with compatibility checks is not optional at any real scale.
Related patterns
Retrieval-Augmented Generation (RAG)
Ground an LLM's answers in retrieved, up-to-date, private documents.
Vector Search + Rerank
Cheaply retrieve a broad candidate set, then precisely re-rank the top results.
Feature Store
Compute features once, serve them consistently to training and inference.
CQRS
Separate models and stores for writes and reads.