Failure detection and health checks

Telling a slow machine from a dead one, quickly and without false alarms.

System design conceptReliability and operations1 Nov 2026

01The intuition

The problem it solves

A load balancer sends every tenth request to server 7. Server 7's disk has just died; it accepts the connection and then hangs. The load balancer does not know. One in ten users now waits 30 seconds and gets an error, and the service's success rate drops to 90 percent while every machine but one is fine.

The naive fix is "ask the server if it is healthy". That misses the hard cases. A hung server may not answer at all, so the check must time out. A slow server answers the check fine while real requests take ten seconds. And a healthy server that paused for one second to collect garbage must not be thrown out for it, because ejecting a healthy machine during a traffic peak makes the remaining ones slower, which makes them look unhealthy, which ejects them too. That cascade has taken down real services.

Build it one piece at a time

Step 1: an active check with a timeout. Every 5 seconds the load balancer calls GET /healthz on each server with a 2-second timeout. A reply of 200 counts as a pass, anything else or silence as a fail.

Step 2: require several failures in a row. Keep a counter per server. Three consecutive fails mark it unhealthy and remove it from rotation; two consecutive passes bring it back. One lost packet no longer ejects a machine.

Server	Last 5 checks	Consecutive fails	State
6	pass pass pass pass pass	0	healthy
7	pass fail fail fail fail	4	unhealthy, out of rotation
8	pass pass fail pass pass	0	healthy

Step 3: make the check mean something. A /healthz that returns 200 unconditionally only proves the process is up. A deep check tests what the server needs: can it reach its database, is its disk writable, is its request queue below a limit. Keep it cheap, because it runs thousands of times a minute across the fleet.

Step 4: watch real traffic too (passive checks). The load balancer already sees every response. If a server's error rate or latency over the last minute is far worse than its peers, eject it even though /healthz passes. Compare against peers, not an absolute number, so a slow database slows everyone without ejecting anyone.

Step 5: cap how many you eject. Never remove more than, say, 20 percent of a pool. If more than that look unhealthy, the problem is almost certainly shared (the database, the network), and ejecting servers would only overload the rest. Serve degraded from everyone and page a human.

Why it works, with the numbers

Detection time is the check interval times the failure threshold, plus the timeout:

worst-case detection ≈ 3 × 5 s + 2 s = 17 s; during it, 1 server in 10 fails ≈ 10% of requests, so about 1.7 "server-seconds" of errors per 10 servers

Tighter intervals detect faster but cost more checks and more false positives; the three-in-a-row rule turns a 1 percent packet loss into a 0.0001 percent false ejection rate per check. The guarantee is bounded: a dead server stops receiving traffic within the detection window. The honest failure modes are the ones the design above guards against: slow is not dead (hence passive checks), a shared dependency is not a server fault (hence the ejection cap), and the checker itself can be partitioned from healthy servers (hence requiring agreement from more than one checker, or from the servers' own heartbeats, as in presence and heartbeats).

The trade-off you must say out loud

Fast detection and few false alarms pull in opposite directions; pick numbers and defend them. The interviewer's next question is what happens when the checker is wrong: the work a "dead" machine was doing may still be running, so anything it does must be idempotent or fenced with a lease.

Where it shows up in the 100 Google problems

Design a load balancer: active and passive checks, consecutive thresholds, and the ejection cap.
Server health monitoring: the same logic over a million machines, with heartbeats instead of polling and alerts that tolerate one lost packet.
Distributed job scheduler and GPU scheduler: a worker declared dead has its jobs reassigned, which only works because jobs are idempotent.
Distributed file system: the master notices a chunk server missing heartbeats and re-replicates its chunks elsewhere.
Upgrade 5,000 servers: each wave waits for health checks to pass before the next begins.