Handle a fire in a data center

A region goes dark in seconds; keep the service up and the data intact.

Problem 079 of 100Infrastructure and platformL61 Nov 2026

Concepts this design leans on

Multi-region and failover — Running in several places so one disaster is a bad hour, not a bad week.
Replication — Keeping copies so reads scale and a dead machine loses nothing.
Failure detection and health checks — Telling a slow machine from a dead one, quickly and without false alarms.
Back-of-the-envelope capacity planning — Turning users and bytes into machines, with arithmetic an interviewer can follow.
Load balancing — Spreading requests over many servers so no one machine decides your latency.

01The design, in the six-step framework

1. Requirements

Functional.

A consumer service (say a mail or photos product) runs in several data centres. One of them catches fire and is gone within seconds; it will not come back for weeks.
Users must keep reading and writing with at most a short blip; no committed data may be lost.
Afterwards, the service must run at full capacity on the remaining sites until the lost one is rebuilt.
Out of scope: the physical response, insurance, and rebuilding hardware; this is the system and the program plan around it.

Non-functional.

Recovery time objective (RTO): user-visible errors for under 2 minutes; full capacity within 30 minutes.
Recovery point objective (RPO): zero for acknowledged writes; a write we told the user succeeded must survive.
Scale: 300 million daily users, 200,000 rps at peak, 50 PB of user data.
Sites: three regions, each sized so any two can carry peak load (N+1 at region granularity).

Back of the envelope

Peak 200,000 rps across three regions is about 67,000 rps each at normal times. If one region dies, the other two must carry 100,000 rps each, so every region must be provisioned at 150 percent of its share, not 100 percent.

Required per-region capacity = peak ÷ (N − 1) = 200,000 ÷ 2 = 100,000 rps

Data: 50 PB with three copies in three regions is 150 PB of disk. Losing a region leaves two copies; re-replicating 50 PB into a new site at 100 Gbit/s takes 50 × 10¹⁵ × 8 ÷ 10¹¹ ≈ 4 × 10⁶ s, about 46 days, which is why the plan starts rebuilding the third copy immediately and why a fourth, cheaper copy (erasure coded) is worth discussing.

2. Core entities

Region: id, state (serving, draining, dead), capacity, health score.
Replica set: data shard, member regions, current leader, commit index.
Traffic policy: DNS and anycast weights per region, the version in force.
Write record: id, shard, committed_at, replication acknowledgements.
Runbook action: step, owner, started, completed; the incident log.

3. API

The user-facing API is unchanged; the design is in the control plane.

POST /regions/{id}/drain and POST /regions/{id}/mark-dead: operator or automation; shifts traffic weights and tells replica sets to stop waiting for that region.
GET /regions/health → per-region error rate, latency, and replication lag, the inputs to the decision.
POST /traffic/policy {weights} → new version, applied by the global load balancer and DNS within 60 seconds.
POST /replicasets/{shard}/reconfigure {members} → add a replica in a surviving region.

4. High-level design

The first five minutes:

Health probes from the edge and the other regions stop getting answers from Region C. Within 30 seconds the failure detector marks it unhealthy; the global load balancer stops routing to it (anycast withdraws the route, DNS weights go to zero).
Every consensus group that had its leader in C elects a new leader among A and B. Writes stall for a few seconds per group, then resume with a two-of-three quorum that no longer needs C.
Autoscaling in A and B absorbs the extra load; the pre-provisioned headroom is what makes this a non-event rather than a cascade.
The failover controller records C as dead, which turns off every "wait for C" path (replication acks, cross-region queues) and starts the re-replication plan.
Operators confirm, open the incident, and begin the program: capacity, data durability, communication, rebuild.

5. Deep dives

Why is no committed write lost, and what does "committed" have to mean?

Because a write was never acknowledged until it was durable in at least two regions. Metadata (mailboxes, albums, indexes) lives in consensus groups spanning the three regions; a commit needs a majority, so it is always on at least two sites. Blobs are written to two regions synchronously and the third asynchronously, and the acknowledgement waits for the two. The cost is write latency: a cross-region round trip of 20 to 60 ms on every write. The alternative, acknowledging after one region and replicating later, is faster and loses whatever was in flight when the fire started. The RPO of zero is a latency decision made long before the incident, and the strong candidate says so.

How do the surviving regions carry the load without becoming the next outage?

Capacity planning at N+1 region granularity: each region always runs at under two-thirds of what it could carry. During the failover, autoscaling adds servers, but databases and caches do not scale in minutes, so the failover also sheds: lower-priority traffic (background sync, recommendations, thumbnail regeneration) is paused by a global flag so the headroom goes to interactive requests. Caches in A and B are cold for users that were homed in C; expect a read-latency bump for ten to twenty minutes and a replica read spike, and pre-warm the most-used keys from the write log if the product cares.

Choice	Steady-state cost	Behaviour when a region dies
Three regions at 150 percent of share	1.5× compute	Two regions carry everything; no shedding needed
Three regions at 100 percent of share	1.0× compute	Shed a third of traffic for an hour while scaling
Two regions, synchronous replication	1.0× compute, 2× data	Survives one loss with zero RPO but no quorum for consensus; needs a witness site

How do we get back to three copies before the next failure?

Two copies of 50 PB is one disk failure from data loss on any given object. The controller immediately adds a third member to every consensus group in the healthiest spare location (a fourth region or an expanded A or B) and starts background re-replication of blobs, rate limited so it does not starve user traffic: at 30 percent of a 100 Gbit/s link the 50 PB takes about five months, which is too long. The real answer is that the third copy should already exist somewhere, as an erasure-coded cold copy that costs 1.5× rather than 3× in disk, so re-replication is a restore, not a rebuild. Erasure coding across regions is the usual compromise between durability and cost at this scale.

What is the program side, if the interviewer is a TPM or EM?

A fire is also a people problem: an incident commander, a communications lead (status page within 15 minutes, honest about degraded features), a capacity lead (purchase and rack hardware for the rebuild or a new region, lead time of months), a data lead (verify every shard has two healthy copies, prioritise the third), and a review within a week that produces the list of single-region dependencies that were discovered the hard way. The plan should name what was assumed and was not true, which in real incidents is usually a service that quietly had one copy of something.

6. What a strong answer sounds like

Open with RTO and RPO as numbers and say that zero RPO is bought with synchronous cross-region writes, which cost latency every day.
Show the N+1 arithmetic: with three regions, each must carry half of peak, not a third.
Explain failover as automation that is rehearsed, with humans confirming, not humans typing commands at 3 a.m.
Do the re-replication maths and reach the conclusion that a pre-existing erasure-coded copy is the only way to be back to full durability quickly.
Expected follow-ups: how to avoid a split brain if C is only partitioned, not burnt (majority quorum, C's groups cannot commit alone); how to test this (regular region evacuation drills); how to handle services that were secretly single-region.
Common mistake: describing backups. Backups restore yesterday; the question is about losing nothing today.

02Where this question was reported

IGotAnOffer · Given as a managerial-TPM program-level prompt (ex-Google Head of TPM)
IGotAnOffer · Google TPM list