Presence and heartbeats
Knowing who is online and when someone silently vanished.
01The intuition
The problem it solves
Your chat app shows a green dot next to friends who are online. How does the server know? When Asha opens the app she tells the server "I am here". When she closes it she could say "I am leaving". But phones do not say goodbye: the battery dies, the train enters a tunnel, the app is killed by the operating system. If the server only believes what it is told, Asha stays green for days after her phone died.
The same question appears inside a data centre. A master hands work to 1,000 worker machines. A worker that catches fire does not send a message saying so. The master must discover the silence on its own, quickly enough that the work is reassigned, but not so hastily that a worker that paused for a second is declared dead and its work done twice.
Build it one piece at a time
Step 1: a heartbeat. Every client sends a tiny "still here" message on a fixed schedule, say every 10 seconds. The server records the time of the last one it received for each client.
| User | Last heartbeat seen | Now | Age |
|---|---|---|---|
| Asha | 12:00:05 | 12:00:12 | 7 s |
| Ravi | 11:58:40 | 12:00:12 | 92 s |
Step 2: a timeout. A client whose last heartbeat is older than a threshold is treated as gone. Pick the threshold as a multiple of the interval: with heartbeats every 10 seconds, a 30-second threshold tolerates two lost messages before declaring a disconnect. Ravi, at 92 seconds, is offline; Asha is online.
Step 3: store it somewhere cheap. A million users sending a heartbeat every 10 seconds is 100,000 writes a second, far too many for a relational database but trivial for an in-memory store. Write the timestamp into a key-value store with an expiry equal to the threshold: the key simply vanishes when the user stops heartbeating, and "is Asha online" becomes "does her key exist". No sweep over all users is ever needed.
Step 4: tell the people who care. Rather than every friend polling, publish a presence change ("Asha went online") to a topic, and the servers holding the sockets of Asha's friends forward it. Status changes are rare compared with heartbeats, so this fan-out is cheap.
Step 5: separate "slow" from "dead" for machines. For workers in a cluster, use two thresholds. After one missed interval, mark the worker suspect and stop giving it new work. After several, mark it dead and reassign its existing work. A worker that was merely paused by garbage collection recovers from suspect without anything being redone.
Why it works, with the numbers
The timeout is a bet between two errors. A short timeout detects real failures fast but turns a brief network hiccup into a false "offline". A long timeout is calm but slow. If heartbeats are sent every i seconds and the network loses a message with probability p, the chance that k consecutive heartbeats are lost by accident is pk. With p = 0.01 and a threshold of 3 intervals:
false offline rate per check = 0.013 = one in a million, detection delay ≤ 3 × 10 s = 30 s
The cost of presence is dominated by heartbeats, so the interval sets the bill: a million users at one heartbeat per 10 seconds is 100,000 small writes a second, which one in-memory store handles. The guarantee is bounded: "a vanished client is noticed within the threshold", never "noticed instantly". The honest failure mode is a partition: if the network between a healthy client and the server breaks, the server sees silence and declares the client dead even though it is fine. Every design that acts on "dead" must tolerate being wrong, which is why work is reassigned with idempotent tasks.
The trade-off you must say out loud
Faster detection costs more heartbeats and more false alarms; calmer detection costs staleness. State the interval and the threshold as numbers and say why. The interviewer's follow-up is "what does the master do when it wrongly declares a worker dead and that worker keeps running", which leads to fencing tokens and leases from distributed locks.
Where it shows up in the 100 Google problems
- Design a chat app: heartbeats over the socket, expiring keys for online status, and a publish of each change to friends' connection servers.
- Design server health monitoring and failure detection: a million machines heartbeating to a sharded collector, with suspect and dead thresholds and alerts that do not fire on one lost packet.
- Design a distributed file system uses the same heartbeat from chunk servers to the master to decide when to re-replicate data.