Safe rollouts
Canaries, waves and instant rollback.
01The intuition
The problem it solves
You have a new version of the checkout service and 1,000 servers running the old one. The fast way is to replace all 1,000 at once. If the new version has a bug that passes tests but fails on real traffic, say it rejects a card type the tests never used, then every checkout in the world fails at the same moment, and fixing it means another deploy of 1,000 servers while the phones ring.
The question is not whether bugs will reach production; some will. The question is how many users see one, for how long, and how fast you can undo it.
Build it one piece at a time
Step 1: a canary. Deploy the new version to one server, or 1 percent of the fleet, and send it real traffic. Compare its error rate and latency with the other 99 percent for 15 minutes. If it is worse, stop and roll that one server back. At most 1 percent of users saw the bug, for at most 15 minutes.
Step 2: waves. If the canary is healthy, continue in increasing steps: 1 percent, 5, 25, 50, 100. Each wave waits for health checks and metrics before the next starts. A bug that shows only under load tends to appear by the 25 percent wave.
| Wave | Servers on new version | Bake time | Users exposed if bad |
|---|---|---|---|
| canary | 10 (1%) | 15 min | 1% |
| 2 | 50 (5%) | 30 min | 5% |
| 3 | 250 (25%) | 1 h | 25% |
| 4 | 1,000 (100%) | done | 100% |
Step 3: automatic rollback. A human watching dashboards is slow and sleeps. Define the abort rule as numbers: if the new version's error rate exceeds the old version's by 0.5 percentage points, or its p99 is 20 percent higher, over a 5-minute window, roll back without asking. Rollback means pointing traffic at the previous version's servers, which still exist because nothing was deleted until the rollout completed.
Step 4: blue-green for the instant switch. Keep two identical fleets, blue (old) and green (new). Deploy to green while blue serves, test green with a trickle, then flip the load balancer. Rollback is flipping back, in seconds. It costs double the capacity during the deploy, so it is used where instant undo matters more than money.
Step 5: separate code from behaviour with flags. Ship the new code with the feature switched off, then turn the feature on for 1 percent of users through a feature-flag service, then more. Now a bad feature is turned off in seconds without any deploy, and the rollout of code and the rollout of behaviour are two independent, reversible dials.
Why it works, with the numbers
Exposure is the sum over waves of the fraction of users on the new version times how long a bug could go unnoticed there. With the plan above and an abort that fires within 5 minutes of a wave starting, a bug caught at the canary costs
1% of users × 5 min = 0.05 user-minutes per 100 users, versus 100% × 15 min for a big-bang deploy, about 300 times less damage
The guarantee is "a regression affects a bounded slice of users for a bounded time and is reversible". The honest limits: a canary cannot catch a bug that only appears in 0.1 percent of requests within 15 minutes of 1 percent of traffic, so slow-burn bugs still reach later waves; two versions run at once, so data written by the new one must be readable by the old one during the rollout and especially during a rollback, which means every schema and message change goes out in a backward-compatible step first; and rolling to every region at once reintroduces the correlated failure that multi-region design was meant to avoid, so regions roll in sequence too.
The trade-off you must say out loud
Safety costs time: a careful rollout takes hours, a big bang takes minutes. For 1,000 servers a day of slow, boring deploys is the price of never having the bad morning. The interviewer's next question is "what about the database migration that goes with this release", where the answer is expand-then-contract: add the new column, deploy code that writes both, backfill, deploy code that reads the new one, and only then drop the old.
Where it shows up in the 100 Google problems
- Global code deployment pipeline: waves across regions with bake times, automatic abort on metrics, and artefacts distributed before the switch.
- Upgrade 5,000 servers without downtime: the same waves applied to kernels and hosts, draining each server from the load balancer first.
- Global configuration and feature-flag service: the second dial, percentages of users rather than percentages of servers.
- Distribute large ML model files: a new model is a rollout too, canaried on a slice of traffic with quality metrics as the abort rule.
- Migrate live users from NoSQL to SQL and monolith to microservices: expand-then-contract over months instead of hours.