Retry with Backoff & DLQ
Retry transient failures with backoff, quarantine what keeps failing.
StarterMessagingPlatform
A consumer retries a failed message with exponential backoff up to a limit, then moves it to a dead-letter queue instead of blocking the main queue or looping forever, with alerting on anything that lands there.
When to use it
- Downstream failures are often transient (throttling, brief outages)
- A handful of bad messages should not block or endlessly retry against the main queue
Trade-offs
- Dead-lettered messages need an operational runbook or they pile up unseen
- Backoff adds latency to the messages that do eventually succeed
Components used
Serverless FunctionMessage QueueContainer ServiceAlerting & On-call
How it works
- Failed messages are retried with exponentially increasing delays plus random jitter, so transient faults get a chance to clear.
- A maximum attempt count bounds the effort. Messages exceeding it are moved to a dead-letter queue rather than retried forever.
- The DLQ becomes an inspectable quarantine — poison messages stop blocking the main queue but are not silently discarded.
Used in the wild
- Any consumer calling a network dependency that may be briefly unavailable.
- Protecting a recovering downstream service from a retry thundering herd.
- Isolating malformed messages that will never succeed regardless of retries.
Good to know
- Jitter is the part people omit and then regret. Without it, all failed messages retry in perfect lockstep and re-hammer the dependency at exactly the same instants.
- A DLQ with no alerting is a silent data-loss mechanism. Messages arrive, nobody looks, and retention quietly deletes them.
Related patterns
Model Serving with A/B Testing
Route inference traffic across model versions to compare live performance.
CQRS
Separate models and stores for writes and reads.
Change Data Capture
Stream a database's row-level changes without touching app code.
Blue-Green Deployment
Run two full production environments, switch traffic between them.