Load balancing
Spreading requests over many servers so no one machine decides your latency.
01The intuition
The problem it solves
A bank branch has one cashier. On the first of the month 300 people arrive to collect pensions. If each visit takes 2 minutes, the last person waits 600 minutes, ten hours. The branch does the obvious thing: it opens five counters and puts a person at the door who says "counter 3 is free, go there".
A web service is the same. One server can answer perhaps 2,000 requests a second. When a million users arrive, you need hundreds of servers, and something at the door that sends each request to one of them. That something is the load balancer. Without it, every client would have to know every server's address, and a server that died would keep receiving requests until someone noticed.
Build it one piece at a time
Step 1: one address in front. Clients connect to a single name, api.example.com. Behind it sits the balancer, and behind the balancer a list of servers.
| After step | Balancer knows | Sends request to |
|---|---|---|
| 1 | servers A, B, C | A, then B, then C, then A (round robin) |
| 2 | A, B, C and whether each answered a check in the last 5 s | only the healthy ones |
| 3 | plus each server's open connections | the one with the fewest |
Step 2: health checks. Every few seconds the balancer sends each server a tiny request. Two failures in a row and the server is taken out of rotation; two successes and it is back. This is what turns "a server crashed" from an outage into a non-event. The failure detection article covers the timing choices.
Step 3: smarter choice. Round robin ignores that some requests are heavy. "Least connections" sends the next request to the server with the fewest in flight; "least response time" prefers the one answering fastest. For servers that keep per-user state in memory, "sticky" routing hashes the user id so the same user lands on the same server.
Step 4: layers. A layer-4 balancer looks only at IP and port and forwards packets; it is very fast and knows nothing about the request. A layer-7 balancer reads the HTTP request, so it can route /video/* to the video servers and /api/* to the API servers, end TLS, and add headers. Big sites use both: layer 4 at the front, layer 7 behind it.
Step 5: no single point of failure. The balancer itself can die. Two balancers share a floating address, or DNS hands out several balancer addresses, or (at Google scale) routers spread packets across many balancers with anycast. The state now is: many clients, a few balancers, many servers, and nothing that stops the site when it dies alone.
Why it works, with the numbers
Capacity adds up. If one server handles 2,000 requests a second at 50 ms each, then n healthy servers handle about 2,000n. To serve 150,000 requests a second with 30 percent headroom:
servers needed = 150,000 ÷ 2,000 × 1.3 ≈ 98, so run 100 and lose a few without noticing
The guarantee is that a request reaches a healthy server and that losing any one machine, balancer included, changes nothing a user can see. The failure modes: a health check that only tests "is the process alive" passes a server whose database connection is broken; sticky sessions pile users onto one server; and a slow server that still answers health checks draws its full share of traffic while timing out on real requests. Least-response-time routing and outlier detection exist for exactly that case.
The trade-off you must say out loud
Every hop costs a little latency and a lot of simplicity. A layer-7 balancer that ends TLS, inspects requests and keeps sticky maps is a small service in its own right, with its own capacity limits. Say what you put where: layer 4 for raw throughput, layer 7 where routing by path or header earns its keep. The next question is nearly always "what happens when the balancer dies?", and the answer is redundancy at the DNS or anycast layer, not a bigger balancer.
Where it shows up in the 100 Google problems
- Design a load balancer: the whole question; the deep dives are health checks, algorithms and making the balancer itself redundant.
- Design a high-scale API gateway: a layer-7 balancer grown up, with auth and throttling at the door.
- Upgrade 5,000 servers without downtime: drain a server from the balancer, upgrade it, add it back; the balancer is what makes zero downtime possible.
- Scale to a 10x traffic surge: the balancer is where new servers are added and where excess load is shed.
- Design LLM serving infrastructure: routing by model and by GPU queue depth instead of plain round robin.