Scale a web application to handle a 10x traffic surge

01The design, in the six-step framework

1. Requirements

Functional.

• An existing web application (a product page, a search box, a logged-in account area) must keep working when traffic jumps to ten times its normal level within minutes.
• The surge may be good news (a launch, a TV mention) or bad news (a competitor outage, a bot wave); the system should tell them apart well enough to act.
• Out of scope: rewriting the application; we have hours to prepare, not months.

Non-functional.

• Normal load 10,000 requests per second; surge 100,000 rps for up to a few hours.
• Read latency p95 under 300 ms during the surge; writes (checkout, sign-up) may queue but must not be lost.
• Availability: the home page and product pages stay up for everyone; degraded features are acceptable, a blank page is not.
• Cost: the extra capacity is temporary and should scale back down.

2. Core entities

• Page / resource: URL, cacheability class (public, per-user, never), TTL, last_modified.
• Session: session_id, user_id, stored outside the app servers so any server can serve any user.
• Write job: job_id, type (order, sign-up, review), payload, idempotency key, state.
• Capacity unit: server group, current count, target count, health.

3. API

• The public API does not change. What changes is the headers it emits: Cache-Control: public, max-age=60, stale-while-revalidate=300 on product and listing pages; private, no-store on account pages.
• POST /orders {idempotency_key, cart} → 202 {job_id} during a surge (queued) instead of 201 (done); the client polls GET /orders/jobs/{job_id}.
• Internal: GET /health on every server, returning 503 when the server is shedding so the load balancer stops sending it traffic.

4. High-level design

A product-page request during the surge:

1. The CDN serves the page from its edge if it is under 60 seconds old; most requests end here.
2. On a miss, the CDN forwards one request to origin and holds the other requests for the same URL until it returns (request coalescing), so a stampede on one hot page becomes one origin call.
3. The load balancer sends the request to a healthy app server; servers that report overload are skipped.
4. The server reads the catalogue entry from the shared cache; on a miss it reads a replica and fills the cache with a TTL plus jitter.
5. A checkout is validated, written to the queue with an idempotency key, and acknowledged as accepted. Workers drain the queue into the primary at a rate the primary can sustain.

5. Deep dives

Where does the ten-times read load actually go?

Into caches, at three layers, each catching what the one above missed. The CDN takes anonymous page views and all static assets, which during a launch is the bulk of traffic. The shared cache takes the catalogue and listing data for logged-in pages that cannot be served from the CDN whole. The database's read replicas take the rest. The subtle part is stampede control: when a hot key expires, a thousand servers miss at once and all hit the replica. Fixes are to serve stale while one request refreshes, to add random jitter to TTLs so keys do not expire together, and to lock the refresh of a key so only one server does it. Without these, the caches are fine at 10× and fall over at exactly the moment of a popular expiry.

How do we make the application servers scale without a rewrite?

Only if they are stateless. The first audit question is where sessions live; if they are in server memory, a user is pinned to one server and adding servers does not help that user. Moving sessions to the shared cache (or a signed cookie) is a small change with a large payoff. Then autoscaling on CPU and request latency can take the group from 50 to 150 servers in minutes, provided the images start fast and the configuration is pulled, not baked in. Pre-warming matters: scale to the expected level before the TV spot, because scaling reacts to load that has already arrived.

What protects the database, which cannot be scaled in an hour?

The primary takes writes only, and writes are the part we can delay. Putting checkout behind a queue changes the user-visible contract from "your order is placed" to "your order is accepted", which the product must agree to, but it turns a 10× spike into a steady drain at the primary's safe rate. The queue must be durable, the job idempotent, and the client must have a way to find out the result. Reads scale with replicas; if replica lag grows, the application can show slightly stale catalogue data but must read a user's own fresh writes from the primary (read-your-writes), which is a per-request routing rule, not a new database.

What do we do when even that is not enough?

Load shedding in priority order. Each server tracks its own queue depth and latency; past a threshold it returns 503 for the lowest-priority requests first (recommendations, reviews, analytics beacons), then for anonymous browsing, and keeps checkout and sign-in alive longest. A static "we are busy, here is the product you wanted" page from the CDN is better than a timeout. If the surge looks like abuse (one client address at 500 rps, no cookies, identical user agents), rate limiting at the edge removes it before it costs anything. Distinguishing good from bad traffic is a dashboard question: ratio of new sessions, conversion rate, and the distribution of requests per client.

6. What a strong answer sounds like

• Do the arithmetic that shows the database is the bottleneck, then show that caches turn 10× into about 2× at origin.
• Ask what is cacheable per page type and what the product will accept as stale or delayed; the answer shapes the design more than any component does.
• Name stampede control explicitly; it is the difference between a cache that helps and one that fails at the worst moment.
• Describe shedding as a priority list and say which requests die first.
• Expected follow-ups: how to rehearse (load test against production-like staging, game day); what to watch during the event (origin rps, cache hit ratio, replica lag, queue age, 5xx rate); how to scale down afterwards.
• Common mistake: starting with sharding the database, which is months of work, when the question allows hours.

02Where this question was reported

• interviewkickstart.com · 'design Google.com so it can handle 10x more traffic'
• scaleengineer.com · Asia: 'handle a sudden increase in traffic to a web application'
• interviewkickstart.com · Practice question
• Glassdoor · Glassdoor question page; generic prompt