Retries, timeouts and backoff
Retrying without creating the storm that takes the service down.
01The intuition
The problem it solves
You call a friend and the line is busy. You call again at once, and again, and again, ten times a second. If a hundred people do that to the same friend, the line never clears, because the retries are now most of the calls. Networks fail in exactly this way: a service hiccups for a second, every client retries immediately, the service receives three times its normal load while recovering, falls over again, and the retries keep it down. This is a retry storm, and it turns a one-second blip into an hour-long outage.
Not retrying is worse: a single lost packet would fail a user's payment. The answer is to retry the right requests, at the right pace, and to give up at the right time.
Build it one piece at a time
Step 1: a timeout on every call. A request with no timeout can wait forever and hold a thread forever. Set one slightly above the dependency's normal slow case: if its p99 is 200 ms, time out at 500 ms. Timeouts are what make failures visible quickly enough to retry.
Step 2: retry only what is safe and worth it. A timeout or a 503 may succeed on a second try; a 400 (bad request) never will, and a 500 from a service that has already done the work may double it. Retry only idempotent requests, or carry an idempotency key so the retry is harmless.
Step 3: wait longer each time. Exponential backoff: wait 100 ms, then 200, then 400, then 800. Three retries over 1.5 seconds give a hiccup time to pass; a fourth immediate retry would not.
Step 4: add jitter. If 10,000 clients all fail at 10:00:00.000 and all back off 100 ms, they all retry at 10:00:00.100, as one wave. Randomise each wait, say anywhere between 0 and the backoff value, so the wave spreads into a trickle.
| Attempt | Backoff cap | Actual wait (random) | Load on the service from 10,000 clients |
|---|---|---|---|
| 1 (fail) | — | — | 10,000 in 1 ms (the original burst) |
| 2 | 100 ms | 0–100 ms | spread over 100 ms, 100 per ms |
| 3 | 200 ms | 0–200 ms | spread over 200 ms, 50 per ms |
| 4 | 400 ms | 0–400 ms | 25 per ms |
Step 5: a budget, and a deadline that travels. Limit retries to a fraction of normal traffic, say 10 percent: if more than one request in ten is a retry, stop retrying until things calm down. And pass the user's deadline down the chain: if the user will give up at 2 seconds, a service 1.8 seconds in should not start a retry that takes 500 ms. Each hop retries within the time it has left, not on its own schedule, which also stops the multiplication of retries across layers (3 retries at each of 3 layers is 27 attempts).
Why it works, with the numbers
With a retry budget of 10 percent, the worst extra load a storm can add is one tenth:
max load = normal × (1 + budget) = 1.1 × normal, instead of normal × (1 + retries) = 4 × normal with 3 blind retries
and with exponential backoff the total time a client spends on a failing call is bounded by the sum of the waits, about 2 × the last wait. The guarantee is that transient failures are hidden from users while a dependency that is truly down receives no more than a bounded extra load. The failure modes are retrying non-idempotent calls (duplicate side effects), retrying inside retries (the multiplication), and timeouts set so long that the user leaves before the first retry.
The trade-off you must say out loud
Every retry trades latency and dependency load for success rate. Say your timeout, your maximum attempts, your backoff and jitter, which status codes you retry, and your retry budget. Say that the user's deadline is propagated. The follow-up is "what when the dependency is down for ten minutes", and the answer is the circuit breaker, which stops the attempts entirely.
Where it shows up in the 100 Google problems
- Slow and unreliable APIs: timeouts, budgets and deadlines are the first deep dive.
- Notification system: APNs and FCM fail transiently; backoff with jitter per device avoids a storm after an outage.
- 100 million emails on five machines: mail servers rate-limit you; retry schedules decide whether the batch finishes tonight.
- Cross-service transactions: each saga step is retried with an idempotency key until it succeeds or is compensated.