Design a high-scale API gateway

Ten million requests a second: auth, routing, throttling and observability at the door.

Problem 063 of 100Infrastructure and platformL61 Nov 2026

Concepts this design leans on

Load balancing — Spreading requests over many servers so no one machine decides your latency.
Rate limiting — Token buckets, sliding windows, and enforcing them across many servers.
Authentication and authorization — Proving who you are, and checking what you may do, on every request.
Observability: metrics, logs and traces — The three signals that explain why a request was slow.
Caching strategies — Read-through, write-through, write-back, TTLs and invalidation.

01The design, in the six-step framework

1. Requirements

Functional

One entry point for all public APIs: authenticate the caller, authorise the call, route to the right backend version, and shape the response.
Rate limiting and quotas per API key and per plan; request validation against a schema.
Protocol translation: HTTPS and gRPC in, HTTP or gRPC to backends; request and response transforms (headers, field masks).
Observability for every call: metrics, logs, traces, and a per-key usage record for billing.
Self-service for API teams: register an API, publish a version, set policies, without touching the gateway's code.
Out of scope: the developer portal UI, billing invoices.

Non-functional

Scale: 10 million requests a second at peak across 5 regions, 2,000 backend services, 50 million API keys.
Latency: under 5 ms p99 added by the gateway, excluding the backend.
Availability: 99.99 percent; the gateway must not fail because its own dependencies (auth store, rules store) are slow.
Safety: a bad policy or a bad backend version must be rollable back in under a minute.

Back of the envelope

10⁷ req/s ÷ 5 regions = 2 × 10⁶ req/s per region; at 20,000 req/s per gateway core, 100 cores, so ~50 two-socket machines per region with headroom

Auth per request from a store: 10⁷ lookups/s is impossible; keys and tokens must be validated locally from cached material

Logs: 10⁷ × 500 bytes = 5 GB/s raw; sample bodies, keep one summary record per call (~100 bytes → 1 GB/s)

2. Core entities

API: api id, owning team, versions, base path, backend targets per version.
Route: match (host, path pattern, method), target (service, version), policies attached.
Policy: auth requirements, rate and quota limits, transforms, timeouts, retry rules; versioned.
Credential: API key or OAuth client, plan, scopes, status; and the signing keys used to verify tokens.
CallRecord: request id, key, route, status, latency breakdown, bytes; the billing and analytics unit.

3. API

Data plane: https://api.example.com/{api}/v{n}/…; the gateway speaks to clients and backends.
PUT /apis/{id}/versions/{v} with routes and backends; POST /apis/{id}/versions/{v}/rollout with a traffic percentage.
PUT /policies/{id}, attachable to an API, a version or a route.
POST /credentials issues a key; POST /credentials/{id}/revoke takes effect within seconds everywhere.
GET /usage?key=&from=&to= for the portal and billing.

4. High-level design

The load balancer spreads connections across gateway nodes in the region.
A gateway node authenticates: an OAuth token is verified locally with cached public keys; an API key is looked up in a local cache of key hashes pushed by the control plane, with a miss going to the credential store.
It matches the route, evaluates the attached policies (scope check, rate limit against local buckets, schema validation), applies transforms, and proxies the call to the backend version chosen by the rollout percentage.
It emits one call record to the telemetry stream and adds trace headers so the backend's span joins the request's trace.

5. Deep dives

How is authentication done at 10 million requests a second?

Never call the identity service on the hot path. Tokens are signed JWTs or opaque tokens exchanged for a short-lived signed form at the edge; verifying a signature is a local CPU operation (about 50 microseconds for an elliptic-curve key) against public keys fetched once an hour. API keys are stored as hashes; the control plane pushes the full set of key hashes with their plan and scopes to every node (50 million × 100 bytes = 5 GB, which fits, or a bloom filter plus a hot cache if it does not). Revocation is a push of the revoked ids within seconds; a token's short lifetime bounds the window for tokens. Authorization is then a scope-to-route check with no external call.

How do rate limits and quotas work across 250 nodes without a round trip?

The same answer as the rate limiter: local token buckets per key, synchronised to a sharded counter store every 100 ms, with keys routed to a consistent subset of nodes to shrink the error. Daily quotas are different: they are enforced from the usage stream with a one-minute lag and a small allowed overshoot, and the gateway blocks a key once the quota service says so. Rules live in the config snapshot, so a plan change is a config push, not a code change.

How do versions roll out and roll back safely?

A version is a set of routes pointing at backend targets. A rollout is a percentage split, decided per request by hashing the request id (or the key, for stickiness) so that the same client sees a consistent version during a session. The control plane validates a snapshot before publishing (every route resolves, no two routes overlap ambiguously, policies parse) and publishes it with a version number; nodes apply snapshots atomically and keep the previous one in memory, so rollback is a single control-plane command that flips a pointer and reaches all nodes in seconds. Gateway nodes keep serving the last good snapshot if the control plane is down; the data plane never blocks on it.

How does the gateway protect itself and the backends?

Per-backend timeouts and connection pools are part of the policy; a slow backend cannot hold gateway threads because the gateway is event-driven and limits in-flight requests per backend, returning 503 early beyond that limit (load shedding) rather than queueing. Retries are allowed only for idempotent methods, once, to a different instance, with a retry budget of 10 percent of traffic. Body size limits, schema validation and header normalisation stop malformed input at the door. For observability, every call record carries the latency split (auth, routing, backend, transform) so a p99 regression is attributable to a stage, and a sampled 1 percent of calls keep request and response bodies for debugging, with PII fields redacted by policy.

How do API teams publish without the gateway team in the loop?

Self-service is a control-plane workflow: a team submits an API definition (an OpenAPI or protobuf schema plus routes and policies) to a registry; the control plane validates it against platform rules (path ownership, naming, required auth, maximum timeouts), runs it against a test gateway, and publishes a snapshot when the checks pass. Ownership of a path prefix is registered once, so two teams cannot claim the same route. Policies are composable and attach at the API, version or route level with clear precedence, and every change is versioned with an author, which makes the rollback story and the audit story the same mechanism.

6. What a strong answer sounds like

Lead with the two numbers that shape everything: 10 million a second means local auth and local limits; 5 ms means no synchronous dependency on the hot path.
Describe config as versioned snapshots pushed from a control plane, applied atomically, with instant rollback and data-plane independence.
Explain token verification with cached keys and API-key hashes pushed to nodes, plus fast revocation.
Show the self-protection: per-backend concurrency limits, retry budgets, idempotency-aware retries.
Follow-ups to expect: multi-region failover (anycast plus per-region capacity), gRPC and streaming (per-message policies, long-lived connections), and WebSocket upgrades (authenticate once, then pass through), and how to add a new policy type (a plugin interface with a resource budget per plugin).

02Where this question was reported

onsites.fyi · L6 sample prompt.