Rate Limiting & Throttling
Cap request rate per client at the edge before it reaches services.
StarterPlatformSecurity
The gateway tracks request counts per client (token bucket or sliding window) and rejects or delays traffic over the limit, protecting backend capacity and giving fair usage across tenants.
When to use it
- A public API must protect itself from abusive or runaway clients
- Different tiers of customer need different usage quotas
Trade-offs
- Overly strict limits cause false throttling of legitimate bursts
- Rate-limit state needs a shared store to work correctly across multiple gateway instances
Components used
Third-party ClientAPI GatewayCacheManaged App Service
How it works
- A gateway counts requests per client key over a time window and rejects those exceeding the allowance, typically with HTTP 429.
- Token bucket is the usual algorithm: tokens refill at a steady rate and each request consumes one, permitting short bursts within a sustained average.
- Enforcing at the edge means rejected traffic never consumes downstream capacity at all.
Used in the wild
- Public APIs enforcing plan quotas.
- Blunting credential-stuffing and scraping without full bot management.
- Protecting an expensive endpoint — search, report generation, LLM inference — from accidental overuse.
Good to know
- Fixed windows allow double the intended rate at boundaries: a client can spend a full allowance at the end of one window and another at the start of the next. Sliding windows or token buckets avoid this.
- Always return Retry-After. Without it, well-behaved clients guess, and their guess is usually to retry immediately.
Related patterns
Model Serving with A/B Testing
Route inference traffic across model versions to compare live performance.
CQRS
Separate models and stores for writes and reads.
Blue-Green Deployment
Run two full production environments, switch traffic between them.
Canary Release
Send a small slice of traffic to the new version before a full rollout.