Database Sharding
Split data across shards by key so no single node holds it all.
AdvancedData
A router directs each request to the shard owning its partition key (e.g. user id range or hash), so total data and write throughput scale beyond what one database node can hold.
When to use it
- A single database instance can no longer hold the data or write volume
- There is a natural partition key with roughly even distribution
Trade-offs
- Cross-shard queries and joins become expensive or impossible without an aggregation layer
- Rebalancing shards as data grows is an operational project of its own
Components used
Managed App ServiceLoad BalancerRelational Database
How it works
- Data is partitioned across independent database instances by a shard key, so each shard holds a disjoint subset of rows.
- A routing layer maps every query to the shard owning that key. Queries that include the shard key are fast; queries that do not must fan out to every shard.
- Each shard can itself have replicas, combining horizontal write scaling with read scaling.
Used in the wild
- Datasets or write volumes that exceed what a single node can hold or absorb.
- Multi-tenant systems sharded by tenant, which conveniently also isolates blast radius.
- Time-series and event data sharded by time or entity id.
Good to know
- The shard key is close to irreversible. Changing it means moving essentially all of your data, which is why the choice deserves far more design time than it usually gets.
- Cross-shard transactions and joins are the tax. Most teams that shard end up denormalising specifically to avoid them.
Related patterns
Retrieval-Augmented Generation (RAG)
Ground an LLM's answers in retrieved, up-to-date, private documents.
Vector Search + Rerank
Cheaply retrieve a broad candidate set, then precisely re-rank the top results.
Feature Store
Compute features once, serve them consistently to training and inference.
CQRS
Separate models and stores for writes and reads.