WebSockets, long polling and SSE

Keeping a connection open so the server can speak first.

System design conceptReal-time and media1 Nov 2026

01The intuition

The problem it solves

Normal web traffic is a question and an answer. The phone asks "any new messages?" and the server replies. The server can never speak first. So a chat app that wants to show a message the instant it arrives has one bad option: ask again and again. Suppose a school chat app with 10,000 students asks every 2 seconds. That is

10,000 students ÷ 2 s = 5,000 requests per second

and almost all of them get the answer "nothing new". Each request carries headers, a TLS handshake or a kept-alive socket, and a database lookup. At a million users it is 500,000 empty questions a second, and a message still waits up to 2 seconds to show. Ask faster and the cost grows; ask slower and the delay grows.

Build it one piece at a time

Step 1: long polling. The phone asks "any new messages?" and the server does not answer immediately. It holds the request open for up to 30 seconds. If a message arrives in that window, the server answers at once and the phone asks again. Empty answers drop from one every 2 seconds to one every 30 seconds, and delivery delay drops to milliseconds. The state after this step: each user has one request parked on some server.

Step 2: server-sent events (SSE). Instead of answering and closing, the server keeps the response open forever and writes a line each time something happens. One request, many answers, one direction. This fits things like stock tickers and build logs where the client only listens.

Step 3: a WebSocket. The phone makes one request that says "upgrade this connection", and after the handshake both sides can send small frames at any time, in either direction, over the same TCP connection. A chat message is now a 50-byte frame, not a 500-byte HTTP request. The state: each online user owns one open socket on one server, and that server knows the user's id.

Step 4: find the right socket. With a million users spread over 100 servers, a message for Ravi must reach the server holding Ravi's socket. Keep a small table, user id to server id, in a shared store such as Redis, updated on connect and disconnect. Sending to Ravi is: look up his server, forward the message there, that server writes to his socket.

Method	Who speaks first	Requests per message	Typical delay	Good for
Polling every 2 s	client	many empty ones	0 to 2 s	rarely changing pages
Long polling	client, server answers late	about 1	milliseconds	simple inboxes
SSE	server	0 after setup	milliseconds	one-way feeds
WebSocket	either	0 after setup	milliseconds	chat, games, live editing

Step 5: keep sockets alive and honest. A phone that walks into a lift drops its socket without saying goodbye. Each side sends a tiny ping every 30 seconds; three missed pongs and the server closes the socket and removes the table entry. On reconnect the client sends the id of the last message it saw so nothing is lost.

Why it works, with the numbers

An idle open socket costs memory, not CPU: roughly 10 to 50 KB of kernel and application buffers. One server with 16 GB of memory can hold

16 GB ÷ 32 KB per socket ≈ 500,000 sockets

so a million online users need only a handful of connection servers, far fewer than the hundreds needed to absorb 500,000 polling requests a second. The guarantee is "a message reaches an online user within one network round trip", typically under 100 ms. The failure modes: a connection server crashing drops all its sockets at once, so clients must reconnect with backoff and jitter or they create a reconnect storm; and load balancers must route the upgrade correctly and allow long-lived connections, which some corporate proxies do not.

The trade-off you must say out loud

Sockets make the connection servers stateful. You can no longer treat them as interchangeable, deployments must drain them gently, and a message for a user must be routed to the one server that holds that user. The interviewer's next question is usually "how do you deliver to a user who is offline", which is where a durable inbox and push notifications take over.

Where it shows up in the 100 Google problems

Design a chat app: one socket per online device, a user-to-server table, and a durable inbox for everyone else.
Design live chat for a YouTube live stream: a million sockets subscribed to one stream, fed by a fan-out tier.
Design Uber: drivers push a location every few seconds over a socket; riders receive the car moving on the map.
Latest stock prices worldwide: SSE or a socket per user, with the server batching ticks into one frame per 200 ms.
Design Google Docs: every keystroke is a socket frame in both directions.