Model serving
Batching, GPUs, caching and streaming tokens behind an SLO.
01The intuition
The problem it solves
A trained model is a file of weights and a function: input in, prediction out. Running it once on a laptop is easy. Running it for a million users at once, each expecting an answer in under a second, is a systems problem. Two facts make it hard. First, accelerators (GPUs and TPUs) are fast only when fed many inputs at a time; one request at a time leaves them mostly idle, and they cost thousands of rupees an hour. Second, a large language model does not answer at once; it produces one token at a time, each needing a pass through the model, so a 300-token reply is 300 sequential steps.
The naive design, one request per model call on a GPU, serves perhaps 5 requests a second per GPU for a large model. A million users at one request a minute need 17,000 requests a second, which is 3,400 GPUs, most of them idle most of the time.
Build it one piece at a time
Step 1: a model server behind a queue. Requests go into a queue; a worker on each GPU pulls from it. The GPU is never waiting for a network call, and a burst of requests becomes a longer queue rather than a crash.
Step 2: batch. The worker waits up to a few milliseconds to gather, say, 32 requests and runs them through the model together. A GPU that did one input in 20 ms often does 32 in 25 ms. Throughput rises about 25×; each request waits a few extra milliseconds.
| Batch size | Time per batch | Requests per second per GPU | Added wait |
|---|---|---|---|
| 1 | 20 ms | 50 | 0 |
| 8 | 22 ms | 360 | ≤ 2 ms |
| 32 | 25 ms | 1,280 | ≤ 5 ms |
Step 3: for language models, batch across time (continuous batching). Replies have different lengths. If a batch waits for its longest reply, short ones hold a slot idle. Instead, when one sequence finishes, a new request takes its slot at the next token step, so the GPU always has a full batch of sequences at different stages. Each sequence keeps its own attention cache (the KV cache) in GPU memory, which is what makes the next token cheap and what limits how many sequences fit.
Step 4: stream the tokens. Send each token to the user as it is produced over a streaming connection. The first token arrives in a few hundred milliseconds and the user reads while the rest is generated; the total time is unchanged but the experience is.
Step 5: cache and route. Identical prompts (a popular question, a system prompt shared by every request) are cached: the KV cache of a shared prefix is reused rather than recomputed. Requests are routed to replicas that already hold the model version needed, with a canary replica for a new model and load balancing by queue depth rather than round robin, because a GPU with a long queue is not helped by more work.
Why it works, with the numbers
Batching trades a small, bounded latency for a large throughput gain, and the gain is what sets the bill. For 17,000 requests a second:
at batch 1: 17,000 ÷ 50 = 340 GPUs busy, but queueing means ≈ 3,400 provisioned for bursts; at batch 32: 17,000 ÷ 1,280 ≈ 14 GPUs busy, ≈ 40 provisioned
The guarantee is an SLO such as "p99 time to first token under 500 ms and 30 tokens per second after that", and it is kept by sizing the fleet so queues stay short and by shedding load when they do not. The honest limits: GPU memory, not compute, usually caps concurrency for language models, because each active sequence holds a KV cache of tens of megabytes; a new model version is gigabytes that must be rolled out to every replica; and a bad input (a 100,000-token prompt) can hog a batch, so inputs are capped and long ones go to a separate pool.
The trade-off you must say out loud
Bigger batches mean cheaper tokens and longer waits; say the batch window and the latency target together. The interviewer's follow-up is how you handle a traffic spike without buying GPUs for the peak: a queue with a maximum wait, priority for interactive traffic over batch jobs, and a smaller fallback model when the queue exceeds a threshold.
Where it shows up in the 100 Google problems
- Design LLM serving infrastructure: continuous batching, KV cache memory as the capacity unit, streaming and prefix caching.
- Retrieval-augmented generation pipeline: an embedding model served in batch for indexing and a generator served interactively, two very different pools.
- Summarise e-commerce reviews: an offline batch pool that maximises throughput and an on-demand pool that minimises latency, with results cached by review set.
- Customer support with escalation and Google Home: interactive serving where time to first token is the number users feel.