Multi-region and failover

01The intuition

The problem it solves

A data centre is a building. Buildings lose power, flood, catch fire and lose their fibre to a digger. If your service lives in one building, each of those events is an outage that lasts until the building recovers, which can be days. Keeping a second copy of the data in another building fixes the "lost forever" part but not the "down for days" part unless the second site can also serve traffic.

There is a second reason. Light takes about 70 ms to cross the Atlantic and back. A user in Mumbai calling a server in Oregon pays 250 ms of pure distance on every request before any work is done. Serving people from a region near them is the only way under 100 ms.

Build it one piece at a time

Step 1: two regions, one active. Run the full stack in region A. Replicate the database continuously to region B, which sits idle except for receiving changes. If A dies, point DNS at B and promote its database. Data loss is whatever had not replicated yet, perhaps a few seconds; downtime is the minutes it takes a human to notice and switch.

Step 2: send users to the nearest region. Use DNS or an anycast front door that answers with the closest healthy region. Stateless services (web servers, caches) run in every region; a user in Mumbai now talks to Mumbai.

Step 3: decide where writes go. This is the hard part. Option one: every region writes to one leader region and reads locally; writes pay the cross-region trip, reads are fast, and there is one truth. Option two: each region accepts writes for its own users (home region) and replicates to the others; fast everywhere, but two regions can change the same record at once and must resolve it. Option three: a consensus-replicated database such as Spanner commits each write to a majority of regions before acknowledging, which costs a round trip but never conflicts.

Step 4: fail over automatically. Health checks from several vantage points decide a region is down; the front door stops sending traffic there; for a leader-based database a new leader is elected in another region. The sequence is tested regularly by actually switching, because an untested failover is a plan, not a capability.

Step 5: run at less than full load everywhere. If two regions each run at 80 percent and one dies, the survivor must take 160 percent and falls over. Three regions at 50 percent each can lose one and run at 75 percent. Capacity, not just copies, is what survives.

Why it works, with the numbers

Availability multiplies when failures are independent. If one region is available 99.9 percent of the time (about 9 hours of outage a year), two independent regions where either can serve give

1 − (0.001)² = 0.999999, about 30 seconds a year, before counting the failover gap itself

The guarantee has two names: RPO, how much data you may lose (seconds of replication lag, or zero with synchronous commits), and RTO, how long you may be down (minutes with manual failover, seconds with automatic). The costs are honest: at least double the infrastructure, cross-region bandwidth, and the engineering of multi-leader conflict handling. The failure modes: a partition that makes both regions think they are the leader (split brain), which consensus or a fencing rule prevents; and correlated failures, such as a bad deploy pushed to every region at once, which no number of regions fixes and only staged rollouts do.

The trade-off you must say out loud

Fast local writes in every region or one consistent truth: you cannot have both without paying a cross-region round trip per write. Say which data needs which. The interviewer's follow-up is what happens to a user whose home region dies mid-session, and whether a shopper in Tokyo can see stock that a shopper in Paris just bought.

Where it shows up in the 100 Google problems

• Handle a fire in a data centre: the whole question is this concept, with the emphasis on capacity headroom and a rehearsed failover.
• Design a load balancer: the global front door that steers users to the nearest healthy region.
• Design Google Meet: media servers in every region, with calls pinned to the region nearest most participants.
• Preferences across Google products: read locally everywhere, write to a home region, replicate with versions.
• Global restaurant menu updates: a write in one region must appear worldwide within a minute, which sets the replication budget.

02Used in these problems

• Design Google Meet
• Design a global restaurant menu update system
• Manage user preferences across many Google products
• Design a load balancer
• Handle a fire in a data center