Upgrade 5,000 servers without downtime

01The design, in the six-step framework

1. Requirements

Short prompt, long follow-ups: "how would you upgrade 5,000 servers without downtime?" The interviewer wants a rollout plan with the arithmetic of capacity, the ordering of steps, and the failure handling, more than a tool choice. Clarify what is being upgraded (OS kernel, a service binary, or both), whether machines are stateless, and what "downtime" means (zero failed requests, or a bounded error budget).

Functional

• Apply an OS and service upgrade to every machine in a 5,000-machine fleet that serves live traffic.
• Keep serving throughout: no request is dropped because a machine was mid-upgrade.
• Verify each machine after upgrade and the fleet after each batch; stop and reverse on trouble.
• Report progress and the version map at any time; finish within a defined window.
• Out of scope: building the images (assume an image and a service binary exist and are tested).

Non-functional

• Capacity: the fleet runs at 60 percent utilisation at peak; the upgrade may take at most 20 percent of machines out at once at peak, more off-peak.
• Time: a machine takes about 12 minutes to drain, reboot into the new image, warm up and pass checks; the whole fleet within 24 hours.
• Safety: an upgraded cohort's error rate must not exceed the control cohort's by more than 0.1 percentage points before the next batch starts.
• Stateful nodes (a few hundred database and cache machines) upgrade with their own rules: never more than one replica of a shard at a time.

2. Core entities

• Machine: id, role (stateless web, stateful shard replica), rack, zone, current and desired versions, health, drain state.
• Batch: ordered set of machines chosen so no shard loses more than one replica and no rack loses more than half its machines.
• Rollout: id, target versions, batch plan, gates, state, operator.
• HealthSnapshot: cohort metrics (errors, latency, saturation) for the upgraded and control groups per batch.

3. API

• POST /rollouts with target image and binary, constraints (max fraction out, per-shard limits, rack spread), gates and time windows.
• POST /rollouts/{id}/pause, /resume, /rollback; GET /rollouts/{id} for batch progress and the version map.
• Machine agent: POST /machines/{id}/drain, /upgrade, /verify, /undrain, each idempotent and reporting a state.
• Load balancer hook: POST /pools/{p}/members/{m}/drain and /enable (the balancer side of the handshake).

4. High-level design

1. The controller reads the inventory and plans batches that respect capacity (at most 20 percent out), rack spread, and the stateful rule (one replica per shard per batch).
2. For a batch: tell the load balancers to drain its members; wait until in-flight requests finish (a timeout caps stragglers); confirm zero traffic.
3. Each machine's agent upgrades: reboot into the new image from the zone mirror, start the new service binary, warm caches, pass local checks; then the controller re-enables it and sends a small share of traffic first.
4. After the batch soaks, cohort health is compared with the rest of the fleet; pass advances, fail pauses, two fails roll back the batch by rebooting into the previous image.

5. Deep dives

How do we guarantee no dropped requests during a machine's upgrade?

The drain handshake. A machine is removed from the balancer's pool before anything stops; new connections stop arriving, and existing requests complete within the drain window (set to the p99.9 request time, say 30 seconds, with a hard cap of 2 minutes). Long-lived connections (streaming, gRPC) are closed gracefully with a go-away signal that tells clients to reconnect elsewhere. Only when the balancer confirms zero in-flight does the agent stop the process. After the upgrade, the machine passes a readiness check that exercises its real dependencies, then receives traffic at a low weight that ramps over a minute so a cold cache or a JIT warm-up does not spike latency. Any step that fails leaves the machine drained, never half in service.

How are batches chosen so capacity and redundancy both hold?

Two constraints decide membership. Capacity: the fraction out is computed from live utilisation, not a fixed number, so batches grow off-peak and shrink if traffic surges mid-rollout; the controller re-plans before each batch. Redundancy: spread each batch across racks and zones so a rack switch failure during the batch does not combine with the upgrade to take out a service, and for stateful machines pick at most one replica per shard, upgrading followers first and a leader only after a handoff. The plan is a greedy fill: sort machines by role, walk shards and racks round-robin, stop when a constraint binds. Show the first batch on the board: 1,000 web machines, 20 per rack, no stateful nodes; batch two adds one replica of each shard.

What gates a batch, and how does the controller avoid false alarms?

Compare the upgraded cohort with the control cohort during a soak of 15 to 30 minutes on error rate, latency and saturation, requiring enough traffic for the comparison to mean something. A fleet-wide incident moves both cohorts together and does not trigger a rollback, but it does pause the rollout, because upgrading during an incident is bad practice. Crash loops and failed readiness are per-machine signals and fail the machine, not the batch, unless more than 2 percent of a batch fails, which halts everything. These are the safe rollout rules, and the honest answer to "how do you know it worked" is the cohort comparison, not green lights on individual boxes (failure detection at the fleet level).

How is rollback done, and what about the kernel?

Two images stay on disk; rollback is a reboot into the previous one plus the previous binary, executed with the same drain handshake and in parallel batches, since the old version is known good. Kernel rollbacks are the same because the image carries the kernel. Data compatibility is the constraint: a service that migrated its on-disk format during the upgrade cannot roll back, so the policy is that upgrades keep formats readable by the previous version for one release, and the rollout tool refuses to start without that attestation. For the stateful tier, rollback follows the same one-replica-per-shard rule, so it is slower; that is acceptable because the stateful tier is small and upgraded last.

6. What a strong answer sounds like

• Start with the capacity arithmetic, derive the batch size, and note that the schedule should lean on off-peak hours.
• Describe the drain, verify, ramp handshake per machine; that is what "without downtime" means operationally.
• Plan batches by two constraints, capacity and redundancy, and treat stateful machines as a special, slower track.
• Gate on cohort-versus-control comparisons with soak time and clear halt rules; keep two images on disk for rollback.
• Follow-ups to expect: a machine that will not come back (replace it from a spare pool and continue), upgrades that change network or config dependencies (order them first), and doing this continuously rather than as an event (a fleet with a small always-on rolling-upgrade budget).
• A common mistake is to answer with a tool name; the interviewer wants the sequence, the numbers and the failure cases.

02Where this question was reported

• Exponent · 'How would you upgrade 5,000 servers?'
• interviewkickstart.com · 'upgrading 5,000 servers without downtime'
• IGotAnOffer · TPM: 'global system to upgrade software on a fleet of machines'
• IGotAnOffer · 'roll out software updates to machines around the world'
• Exponent / Aced · Exponent question DB.
• interviewkickstart.com · Listed under 'real questions asked in real interviews'.