Design a global code deployment pipeline

01The design, in the six-step framework

1. Requirements

The ex-Googler mock version: ship a 10 GB binary to hundreds of thousands of machines in 5 to 10 regions within 30 minutes, every day, safely. Two problems hide inside: distributing big artifacts fast, and rolling them out without taking production down.

Functional

• Build artifacts are published once; machines in every region fetch the version they are told to run.
• Rollouts proceed in waves (canary, then percentages, then regions) with health gates and automatic halt.
• Instant rollback to the previous version, globally, within a minute.
• Operators see exactly which version each machine runs and why a rollout stopped.
• Out of scope: the build system and tests (the CI system), configuration changes (the config service).

Non-functional

• Scale: 100,000 machines, 8 regions, a 10 GB binary, 50 services deploying daily, up to 5 concurrent rollouts.
• Speed: the artifact is available in every region within 10 minutes; a full fleet rollout can complete in 30 minutes when health allows.
• Safety: no more than 1 percent of capacity is unhealthy at any instant due to a deploy; a bad version is detected within one wave.
• Network: inter-region bandwidth use is bounded; intra-region distribution must not saturate top-of-rack links.

2. Core entities

• Artifact: version id, service, content manifest (list of chunk hashes), size, signature, build provenance.
• Chunk: hash, bytes (4 MB); shared across versions.
• Rollout: rollout id, service, target version, wave plan, current wave, state, health gates, author.
• Machine: machine id, region, rack, current version, desired version, health, last report.
• WaveReport: rollout id, wave, machines updated, error rate, latency, verdict.

3. API

• POST /artifacts uploads a manifest and missing chunks; returns the version id once signed and replicated to all regions.
• POST /rollouts with service, version, wave plan, gates; POST /rollouts/{id}/pause, /resume, /rollback.
• Agent-facing: GET /machines/{id}/desired (long-poll or stream) and POST /machines/{id}/report with current version and health.
• GET /rollouts/{id} for wave status and the reason for any halt.

4. High-level design

1. The build publishes a manifest of chunk hashes and uploads only chunks the origin does not already have (content addressing deduplicates across versions).
2. The origin replicates new chunks once to each regional mirror; the registry marks the version available when all regions confirm.
3. The rollout controller sets the desired version on the first wave's machines; their agents fetch missing chunks from rack-local peers, falling back to the regional mirror, verify every chunk hash and the manifest signature, assemble the binary and swap it in with a drain and a health check.
4. Health metrics from updated machines are compared with the rest; the controller advances, pauses or rolls back the wave accordingly.

5. Deep dives

How does a 10 GB binary reach 100,000 machines without melting the network?

Three levels of fan-out. The origin to 8 regional mirrors is 80 GB once. Within a region, machines fetch chunks from peers in the same rack first (a small tracker per region tells an agent which neighbours already have a chunk), so the mirror serves each chunk a handful of times and the rack switch carries the rest at local bandwidth. Chunks are 4 MB and fetched in parallel from several peers, like a torrent but with the manifest signed by the registry. Deltas come free: an agent already holding version N has 95 percent of version N+1's chunks. Rate limits per agent and per mirror keep a deploy from starving production traffic on shared links. This is the CDN idea applied inside the data centre.

How are waves and health gates designed so a bad build is caught early?

The plan: one canary machine per region, then 1 percent, 10 percent, 50 percent, 100 percent, with a soak time per wave proportional to how long problems take to show (memory leaks need an hour; crashes need a minute). Gates compare the updated cohort with the control cohort on error rate, latency and crash loops, not against fixed thresholds, so a site-wide incident does not falsely blame the deploy. Advance requires enough traffic on the cohort to be statistically meaningful; a wave on an idle service soaks longer. Any gate failure pauses and pages; two failures roll back automatically. The 1 percent unhealthy budget is enforced by never updating more than that fraction at once within a region and by draining before swapping (safe rollouts).

How does rollback take a minute when the rollout took thirty?

Every machine keeps the previous version's binary on disk (it is only chunks it already had). Rollback is a desired-version change to the previous version, which agents apply with a fast swap and no fetching, in parallel rather than in waves because the previous version is known good. The controller still drains and health-checks per machine, so even a rollback does not exceed the unhealthy budget; a region of 12,000 machines rolls back in a few parallel batches, under a minute. State persistence matters: schema or data-format changes must be backward compatible for one version, which is a policy enforced in review, or rollback is impossible and the pipeline must say so before the rollout starts.

How is the fleet's state known and trusted?

Agents report their current version and health every 30 seconds; the controller treats a missing report as unknown and excludes the machine from wave arithmetic rather than assuming success (failure detection). Every artifact is signed at build time and verified by the agent before swap, so a compromised mirror or peer cannot inject a binary. Observability is per rollout: a timeline of waves, the metric comparisons that gated them, and the per-machine version map, which is also what an incident responder opens first when something changed at 14:02.

6. What a strong answer sounds like

• Do the petabyte arithmetic and let it force content-addressed chunks, one replication per region and peer fetching.
• Define waves with cohort-versus-control health gates and soak times, and the 1 percent unhealthy budget.
• Make rollback a desired-version flip to a binary already on disk, with the backward-compatibility caveat.
• Insist on signed artifacts verified at the edge and on unknown-until-reported machine state.
• Follow-ups to expect: stateful services (rolling restarts with leader handoff), coordinated multi-service releases (dependency order in the plan), and emergency pushes (a documented break-glass path that skips soak but not canaries).

02Where this question was reported

• Blind · Software release pipeline
• YouTube · YouTube: 'Google Systems Design Interview With An Ex-Googler' (Clement Mihailescu); global fast code deployment, 5-10 regions, 10 GB binaries, 30-min window, ro
• topofmind.dev · Blog notes on the same video; compares pubsub+SQL dispatcher and tiered caching with Clement's GCS-replication answer.