Pipes and Filters
Chain small, single-purpose processing stages behind a stream.
IntermediateMessagingData
Each filter reads from one stream, does one transformation, and writes to the next -- validation, enrichment, aggregation as separate, independently scalable and replaceable stages instead of one monolithic processor.
When to use it
- A data pipeline has distinct transformation steps that change independently
- You want to reorder, insert or replace a single stage without touching the rest
Trade-offs
- Each hop adds latency and another stream to operate
- End-to-end tracing needs correlation ids threaded through every filter
Components used
Event StreamStream ProcessingData Warehouse
How it works
- Processing is decomposed into independent stages, each doing one transformation and knowing nothing about its neighbours.
- Stages are connected by streams or queues, so each can scale, fail and be redeployed independently.
- New behaviour is added by inserting a stage rather than modifying existing ones.
Used in the wild
- ETL and stream enrichment pipelines.
- Media processing chains — decode, transform, watermark, encode.
- Log and event processing where parsing, enrichment and routing are distinct concerns.
Good to know
- Unix pipes are the canonical implementation, and the pattern's design philosophy is essentially Doug McIlroy's: write programs that do one thing well and connect them by streams.
- Every stage boundary is a serialisation cost. Decomposing too finely produces a pipeline that spends most of its time encoding and decoding rather than working.
Related patterns
Retrieval-Augmented Generation (RAG)
Ground an LLM's answers in retrieved, up-to-date, private documents.
Vector Search + Rerank
Cheaply retrieve a broad candidate set, then precisely re-rank the top results.
Feature Store
Compute features once, serve them consistently to training and inference.
CQRS
Separate models and stores for writes and reads.