Design a load balancer

01The design, in the six-step framework

1. Requirements

The prompt ranges from "front-end server selection for google.com" to "a cloud load balancer product". Scope it to a regional layer-4 and layer-7 load balancer in front of a pool of stateless servers, exposed on one virtual IP, with health checking and graceful operations.

Functional

• Accept TCP and HTTP(S) traffic on a virtual IP and distribute it across healthy backends.
• Layer-7 features: TLS termination, routing by host and path, header insertion, HTTP/2 to clients.
• Health checks that remove sick backends within seconds and add them back when they recover.
• Drain a backend gracefully (finish in-flight requests, take no new ones) for deploys.
• Session affinity when a service asks for it.
• Out of scope: global anycast and DNS steering across regions (mentioned in a deep dive), WAF rules.

Non-functional

• Scale: 2 million requests a second, 10 million concurrent connections, 500 backends per pool, thousands of pools.
• Latency: under 1 ms added at layer 4, under 3 ms at layer 7 including TLS.
• Availability: the balancer itself must have no single point of failure; a balancer node failure drops no more than a few seconds of new connections.
• Fairness: no backend receives more than 20 percent above the average load.

2. Core entities

• VirtualService: virtual IP and port, protocol, TLS certificate, routing rules.
• BackendPool: list of backends, balancing policy, health check spec, affinity setting.
• Backend: address, weight, health state, in-flight count, drain flag.
• Connection: 5-tuple, chosen backend, start time; the state a layer-4 node keeps.

3. API

• Data plane: clients connect to the virtual IP; nothing else.
• PUT /pools/{id} with backends, weights, policy and health check; POST /pools/{id}/backends/{b}/drain.
• PUT /services/{id}/routes with host and path rules; PUT /services/{id}/tls for certificates.
• GET /pools/{id}/health for operators and for autoscalers.

4. High-level design

1. The virtual IP is announced by every layer-4 node; routers spread packets across them with equal-cost multipath hashing.
2. A layer-4 node hashes the connection's 5-tuple to pick a layer-7 proxy with consistent hashing, so every node picks the same proxy for the same connection without sharing state, and forwards the packet encapsulated.
3. The layer-7 proxy terminates TLS, matches the host and path to a pool, picks a backend by the pool's policy, and proxies the request.
4. Health checkers probe backends and push state to proxies; the control plane pushes configuration to both tiers within seconds.

5. Deep dives

Why two tiers, and how does the layer-4 tier avoid being a single point of failure?

Layer 4 is cheap and stateless: it moves packets by hashing and never parses HTTP, so a node handles millions of packets a second. Layer 7 is expensive (TLS, parsing) but is where routing and policy live. Splitting them lets each scale on its own limit. The layer-4 tier has no master: every node announces the VIP, routers hash flows across them, and because the node-to-proxy choice is a deterministic consistent hash of the 5-tuple, a layer-4 node dying only reroutes its flows to other nodes that make the same choice; existing connections survive. This is the Maglev idea; naming the property (connection-preserving hashing with no shared state) matters more than the name.

Which balancing policy, and how does it stay fair under uneven backends?

Round robin ignores that requests differ; a slow backend accumulates work. Least-connections at the proxy sends new requests to the backend with the fewest in flight, which adapts to slow machines. Across many proxies, each sees only its own connections, so use power of two choices: pick two backends at random and send to the less loaded, which gets near-optimal balance with no global state. Weights handle heterogeneous hardware. Session affinity, when required, is a cookie or a hash of the client id to a backend, with a fallback when that backend is unhealthy; say that affinity fights balance and should be the exception (load balancing).

How are health and draining done without flapping or dropping requests?

Active checks every 2 seconds against a cheap endpoint, with a backend marked unhealthy after 3 failures and healthy after 2 successes, so a single timeout does not flap it. Passive signals (5xx rate, connection resets) feed an outlier detector that ejects a backend early and re-admits it after a cooldown. Drain sets a flag: proxies stop sending new requests, wait for in-flight ones up to a timeout, then remove it; the deploy tool waits for the proxies to confirm before stopping the process. If every backend fails checks at once, assume the checker is wrong rather than the pool, and keep routing to all (panic threshold); a balancer that removes the whole pool causes the outage it was meant to prevent (failure detection).

What about overload and multi-region?

When backends are saturated, the proxy applies backpressure: a per-pool concurrency limit that returns 503 quickly rather than queueing, with retry budgets so clients do not amplify a problem. Retries go to a different backend, at most once, and never for non-idempotent requests. For multi-region, the same VIP is announced from several regions by anycast; the nearest region serves, and if a region's capacity is exhausted its proxies spill a fraction of traffic to the next region. DNS-based steering is the slower fallback. These are the follow-ups for a cloud product version of the question.

How does configuration reach thousands of proxies safely?

The control plane stores pools and routes in a versioned config store and pushes snapshots to proxies over a streaming channel, so a change is live within a few seconds. Every snapshot is validated before push (no empty pools, certificates that parse, no overlapping routes) and applied atomically on the proxy, so a half-applied config never routes traffic. Rollouts of risky changes go to one percent of proxies first; because the layer-4 tier hashes flows evenly, that canary sees a representative slice of traffic. If the control plane is down, proxies keep serving the last good config; the data plane never depends on the control plane being up.

6. What a strong answer sounds like

• Size by packets and bytes, split the tiers, and explain why the layer-4 tier needs no shared state.
• Choose least-connections or power-of-two choices with weights, and say why round robin is not enough.
• Give concrete health-check thresholds, the panic threshold, and the drain handshake.
• Mention retries with budgets and idempotency as the overload safety.
• Follow-ups to expect: TLS at scale (session resumption, certificate rotation without restarts), long-lived connections (gRPC needs per-request balancing or periodic connection recycling), and observability (per-backend latency histograms exported from the proxies).
• A common mistake is to draw one load balancer box; the interviewer will ask what happens when it dies.

02Where this question was reported

• Hello Interview · 'Design a load balancer' as a Google example
• PracHub · PM: load balancing google.com front ends
• Blind · Friend was asked: design load balancer in cloud
• Taro · Distributed system with load balancers and a caching strategy; explain deadlock vs livelock trade-offs