Backpressure and load shedding
Slowing producers down instead of falling over.
01The intuition
The problem it solves
A water tank fills from a tap and drains through a pipe. If the tap runs faster than the pipe can drain, the tank fills, then overflows, and the floor is flooded. No amount of bigger tanks fixes a tap that is permanently faster than the pipe; it only delays the flood.
A server is a tank. Requests come in from clients at some rate; the server handles them at some capacity. If arrivals exceed capacity, the queue of waiting requests grows, memory fills, every request waits longer, clients time out and retry, which adds more arrivals, and the server falls over. The fall is not graceful: at 110 percent load you do not serve 91 percent of requests, you serve nearly none, because every request now waits so long that all of them fail.
Backpressure is turning the tap down. Load shedding is opening a drain near the top of the tank so some water is lost on purpose and the rest keeps flowing. Both are better than the flood.
Build it one piece at a time
Step 1: bound the queue. Give the server a queue of at most, say, 100 waiting requests. When it is full, reject the 101st immediately with an error that means "too busy", HTTP 503 or 429, instead of accepting it and letting it wait for ten seconds. A fast no is better than a slow nothing.
| Arrival rate | Capacity | Queue | Served | Rejected fast | Latency of served |
|---|---|---|---|---|---|
| 800/s | 1,000/s | near 0 | 800/s | 0 | normal |
| 1,500/s | 1,000/s | 100 (full) | 1,000/s | 500/s | bounded by queue length |
Step 2: tell the producer to slow down. When the consumer is a service you control, the signal travels upstream. In TCP it is the receive window; in gRPC and reactive streams it is "I can take 10 more"; in a queue system it is the consumer lag that the producer watches. The producer buffers briefly, then slows its own source, until the pressure reaches the edge where a human or a client can be told to wait.
Step 3: shed by priority. Not all requests are equal. Tag each with a priority: a payment confirmation is critical, a recommendation refresh is optional. When the queue passes 70 percent, reject optional work; at 90 percent, reject everything but critical. Users see a feed without recommendations instead of a feed that never loads.
Step 4: shed by cost. A request that scans a month of data costs a thousand times a request that reads one row. Measure the cost as the request runs and limit concurrent expensive requests separately, so ten heavy queries cannot starve ten thousand light ones.
Step 5: stop the retry storm. A rejected client that retries immediately adds to the load that caused the rejection. Clients must back off exponentially with jitter, and the server's rejection can carry a Retry-After hint. Without this, shedding only moves the flood by a second.
Why it works, with the numbers
Queueing theory says waiting time explodes as utilisation approaches 1. For a simple queue with utilisation ρ = arrivals / capacity, the average wait in units of service time is
wait ≈ ρ / (1 − ρ): at ρ = 0.5 that is 1, at 0.9 it is 9, at 0.99 it is 99
So a server running at 99 percent is already a hundred times slower than one at 50 percent, and above 1 the wait is infinite. A bounded queue caps the wait at queue length divided by capacity: 100 requests at 1,000 a second is 100 ms, whatever the arrival rate. The guarantee is that the requests you do serve are served within a known time, and that the system survives any arrival rate. The failure mode is that you reject real users, which is why priority and cost-based shedding exist, and why capacity must still be grown afterwards.
The trade-off you must say out loud
You choose to fail some requests on purpose so that most succeed. Say what the client sees (a fast error with a retry hint), what gets dropped first, and where the pressure signal stops. The follow-up is "why not autoscale", and the answer is that new machines take minutes and shedding takes milliseconds; you need both.
Where it shows up in the 100 Google problems
- 10x traffic surge: shedding optional work is how the site stays up while capacity is added.
- Load balancer: the balancer rejects or queues when every backend reports it is full.
- Managed pub/sub: publishers are slowed when a topic's storage or a consumer's lag passes a limit.
- Log pipeline: agents drop debug logs first when the collector is behind.
- Slow and unreliable APIs: bounded concurrency toward the slow dependency keeps your own threads free.