Design LLM serving infrastructure

01The design, in the six-step framework

1. Requirements

Functional.

• Serve a large language model (tens of billions of parameters) behind an API: a prompt in, a stream of tokens out, with optional tool calls and system prompts.
• Support both interactive chat (latency matters) and bulk jobs (throughput matters) from the same fleet.
• Models are versioned; a new version rolls out gradually and can be rolled back; several models may be served at once.
• Out of scope: training, the safety classifiers' internals (we call them), billing.

Non-functional.

• 20,000 concurrent conversations, 2,000 requests per second, average prompt 1,000 tokens, average answer 300 tokens.
• Interactive SLO: time to first token under 500 ms at p95; steady streaming at 30 tokens per second or better per user.
• Throughput: 600,000 generated tokens per second across the fleet at peak.
• Availability 99.9 percent; cost per million tokens is a first-class metric.

2. Core entities

• Request: request_id, tenant, model, messages, parameters (max tokens, temperature), priority class (interactive, batch), deadline.
• Sequence: the in-flight generation state: prompt tokens, generated tokens, KV cache blocks, position in the batch.
• Model version: name, weights location, tokenizer, quantisation, hardware profile, canary status.
• Replica: model version, accelerators, loaded weights, current batch, free KV blocks.
• Usage record: request_id, prompt tokens, generated tokens, latency, replica, version.

3. API

• POST /v1/chat {model, messages, stream: true, max_tokens, tools} → server-sent events, one per token or chunk, then a final usage event. The interactive path.
• POST /v1/batch {model, inputs: file} → job id; results written to a file within hours at a lower price. The throughput path.
• GET /v1/models → versions available to this tenant.
• Internal: router → replica POST /generate {request, deadline}; replica → router heartbeats with free KV blocks and queue depth, which drive routing.

4. High-level design

One chat turn:

1. The gateway authenticates, checks the tenant's rate limit and token quota, runs the prompt through a fast safety classifier, and opens a streaming response to the client.
2. The router picks a replica of the requested model version, preferring one that already holds the KV cache for this conversation's prefix (the system prompt and earlier turns) and has free KV blocks; otherwise the least-loaded replica.
3. The replica admits the request into its running batch at the next decode step (continuous batching): the prompt is processed in one prefill pass, then tokens are generated one step at a time alongside the other sequences.
4. Each generated token is sent back through the gateway as a stream event; the first arrives after prefill, which is what the 500 ms SLO measures.
5. On completion the KV blocks are freed (or kept briefly for a follow-up turn), and a usage record is written for billing and capacity planning.

5. Deep dives

How is the hardware kept busy without hurting time to first token?

Continuous batching: instead of forming a batch and running it to completion, the replica runs one decode step for every active sequence, admits new sequences between steps, and retires finished ones, so the batch is always as full as memory allows and a new request waits at most one step (tens of milliseconds) to join. Prefill of a long prompt is the exception: it is compute-heavy and would stall the decode steps of everyone else, so it is chunked and interleaved, a few hundred tokens per step. The KV cache is managed in fixed-size blocks allocated from a pool (paged attention), which removes fragmentation and lets a replica hold the number of sequences its memory actually allows rather than a pessimistic maximum. These two techniques are the reason the per-replica throughput number above is achievable in practice.

How do interactive and batch traffic share a fleet?

Two queues, one fleet, strict priority with a floor. Interactive requests are admitted first; batch requests fill whatever KV memory and step capacity is left, and are preempted (their KV blocks evicted, to be recomputed later) when interactive demand rises. A floor guarantees batch some progress so jobs finish. This is load balancing plus queueing with priorities, and it is also the cost story: batch traffic turns the fleet's idle capacity at night into paid tokens, which is why it can be priced lower. Admission control protects the SLO: when the interactive queue's wait exceeds a threshold, the gateway returns a retry-after rather than letting every request be slow.

How do we route for cache hits without creating hot spots?

Conversations are stateless at the API (every turn sends the full history) but the replica that handled the last turn still has the KV cache for the prefix; sending the next turn there skips most of the prefill. The router hashes the conversation id to a preferred replica (consistent hashing within a model version's pool) and falls back to the least-loaded one when the preferred is saturated, trading a cache miss for latency. Shared system prompts across a tenant are cached once per replica and hit by every conversation. The replica reports free KV blocks in its heartbeat, so the router's view of load is memory, not request count, which is the right unit here.

How does a new model version roll out?

Weights (hundreds of gigabytes) are distributed to replicas from the registry through a peer-to-peer or regional cache so 900 replicas do not pull from one bucket; a replica loads the new version alongside the old if memory allows, or is drained and reloaded in waves. Traffic shifts by percentage with the usual rollout gates: latency, error rate, and quality evaluations on a fixed prompt set scored automatically, because a model can be fast, up and worse. Rollback is a routing change; the old weights stay resident until the new version has served a day cleanly.

6. What a strong answer sounds like

• State that decoding is memory-bandwidth bound and derive tokens per second per replica from weights size and bandwidth; that single calculation organises the whole design.
• Name continuous batching and paged KV cache as the mechanisms that make batching compatible with low first-token latency.
• Describe priority queues with preemption for interactive versus batch, and admission control to protect the SLO.
• Explain prefix-aware routing and why load is measured in KV blocks.
• Expected follow-ups: tool calls (the model emits a call, the gateway executes it, the result is appended and generation resumes on the same replica); multi-tenant fairness; what to do when a region's accelerators are full (spill to another region at a latency cost).
• Common mistake: treating the model server as a stateless web service behind a round-robin balancer, which throws away the cache and the batching.

02Where this question was reported

• Exponent · Now appears in general SWE loops, not only ML roles
• PracHub · PM: 'Gemini request path: from user prompt to grounded, streamed answer'
• Prepfully · Candidate reports: LLM serving infra under low-latency, high-throughput.
• aie-blog.pages.dev · 'Distributed inference service for a 175B-parameter LLM'; TPU vs GPU, version integrity.
• PracHub · May 2026 Senior+: distribute large ML model files across a serving fleet.