Sharding and partitioning

Splitting data by key so each machine holds a slice it can handle.

System design conceptFundamentals1 Nov 2026

01The intuition

The problem it solves

A library keeps every borrower's record in one register. With 500 borrowers one clerk manages. With 5 million borrowers the register is a room of shelves and one clerk, and every lookup walks the whole room. The fix is old: split by the first letter of the surname, give each range its own room and its own clerk. A borrower named Rao goes to the R room; nobody else is slowed down.

A database on one machine hits the same wall: the data no longer fits on its disk, or the writes arrive faster than one machine can commit them. Sharding (also called partitioning) splits the data by a key across many machines, each holding a slice it can handle on its own.

Build it one piece at a time

Step 1: pick the key. Every record must have a field you can split on. For a URL shortener it is the short code; for a chat app it is the conversation id; for Drive it is the owner's user id. The key decides which room a record lives in, so it must be present on every read.

Step 2: pick the rule. Two common rules:

Rule	How	Good at	Bad at
Range	keys A–F on shard 1, G–M on shard 2, …	scans over a range ("all events in March")	hot ranges: every new id lands on the last shard
Hash	shard = hash(key) mod `n`	spreading load evenly	range scans must ask every shard

Step 3: route the request. A small routing layer (or the client library) computes the shard from the key and sends the query straight there. With 4 shards and hash routing, user 1,234,567 goes to shard hash(1234567) mod 4, every time, from every server.

Step 4: add a shard. With mod n, going from 4 to 5 shards changes the home of almost every key, which means moving nearly all the data. Consistent hashing fixes that so only about one fifth moves. Range sharding instead splits the busiest range in two.

Step 5: deal with hot keys. One celebrity's inbox or one viral video can be busier than a whole shard. Split that key further (append a random suffix and fan reads across the copies), or put it in a cache in front. The state now: a key, a rule, a router, a way to grow, and a plan for the one key that breaks the rule.

Why it works, with the numbers

Sharding turns a limit on one machine into a limit on the total. If one machine can store 2 TB and commit 10,000 writes a second, then 50 shards store 100 TB and commit 500,000 writes a second, as long as the key spreads the load evenly. The arithmetic an interviewer wants to hear:

shards = max( total data ÷ data per shard , peak writes/s ÷ writes per shard/s ) × headroom

For 60 TB and 300,000 writes a second: max(30, 30) × 1.5 ≈ 45 shards. The invariant is "one key, one shard": a record is always found in exactly one place, so single-key reads and writes stay as fast as on one machine. What you lose is anything across keys: a join between two users on different shards, a transaction that touches two shards, or a count over all records now needs a scatter-gather or a separate aggregate.

The trade-off you must say out loud

The shard key is the one decision that is hard to undo, so say why you chose it and which queries it makes awkward. Sharding by user keeps a user's data together but makes "all messages in a group chat" expensive; sharding by conversation does the reverse. The follow-ups are predictable: "what about a hot shard?", "how do you reshard without downtime?" and "how do you do a cross-shard transaction?", and the honest answer to the last one is usually "avoid needing one".

Where it shows up in the 100 Google problems

Design Google Drive: metadata sharded by owner so a user's whole tree sits on one shard.
Design a URL shortener: hash the short code; there is never a range query, so hash sharding is free of downsides.
Design Twitter search: shard the index by time so recent tweets, the ones everyone searches, sit on a few hot shards you can replicate heavily.
Design Bigtable: range sharding by row key with tablets that split when they grow, the textbook case.
Design a globally consistent database: the same splits, plus a way to move them between continents while serving traffic.