Design a service that depends on slow and unreliable APIs

01The design, in the six-step framework

1. Requirements

Functional.

• Our service answers a user request by calling three or four third-party APIs (a payment provider, a shipping-rate service, an address validator) and combining their answers.
• Some of those APIs take two to ten seconds at times and fail outright a few percent of the time; we do not control them.
• The user must get a response, even if parts of it are partial or stale, and must never be charged twice.
• Out of scope: the business logic of each partner; we treat them as black boxes with an SLA we cannot trust.

Non-functional.

• Our own p99 latency under 800 ms, regardless of partner behaviour.
• 5,000 requests per second at peak; each fans out to up to four partner calls.
• Availability 99.95 percent for the read path even when a partner is down.
• Correctness for side effects: a payment is attempted at most once per order.

2. Core entities

• Request: request_id, user_id, order payload, deadline (absolute time by which we must answer).
• Partner call: partner, request_id, idempotency_key, attempt, started_at, outcome.
• Cached answer: partner, cache key (for example a postcode pair), value, fetched_at, ttl.
• Circuit state: partner, state (closed, open, half-open), failure rate over the last window, opened_at.
• Deferred job: request_id, partner, payload, next_attempt_at; used when the answer can arrive later.

3. API

• POST /checkout {order, idempotency_key} → {status: complete | pending, quote, payment_state}. The client supplies the key so a retried submit cannot pay twice.
• GET /checkout/{id} → the same shape; the client polls while status is pending.
• GET /shipping-quote?from=&to=&weight= → {rate, stale: bool, as_of}. Reads may be served stale and say so.
• Internal: GET /partners/{name}/health → circuit state and recent latency percentiles, for operators and for the dashboard.

4. High-level design

A checkout request, end to end:

1. The API stamps the request with a deadline (now + 700 ms) and an idempotency key, and writes an order row in state pending.
2. The aggregator asks the cache for the shipping rate and address validation; hits are returned immediately with their age.
3. Misses go through the partner client in parallel. Each call has its own timeout derived from the remaining deadline, a per-partner concurrency limit (bulkhead) and a circuit breaker.
4. The payment call is made exactly once with the idempotency key; if it times out, we do not retry blindly, we query the payment's status by key.
5. At the deadline the aggregator returns what it has: a complete answer, or a pending one with the parts still outstanding handed to the deferred-job queue.
6. A worker completes the deferred parts with backoff, updates the order, and the client sees it on the next poll or by push.

5. Deep dives

How do we keep our p99 under 800 ms when a partner takes ten seconds?

The naive design waits for every partner. Its p99 is the slowest partner's p99, and during an incident it is the partner's timeout, which is often 30 or 60 seconds. Three mechanisms break the dependence. A deadline is attached to the request and propagated: each outbound call gets a timeout equal to the time left, so nothing can outlive the user's budget. Hedging helps with slow-but-healthy partners: for an idempotent read, if the first call has not answered by the partner's p90 (say 250 ms), send a second one and take whichever returns first; this trades about 10 percent more calls for a much flatter tail. And a stale-while-revalidate cache turns most reads into local answers: serve the rate we fetched four minutes ago, mark it stale, and refresh it in the background. The honest statement to the interviewer is that we cannot make a partner fast; we can only bound how long we wait and decide what to show when the bound hits.

How do retries avoid making the outage worse?

A partner that is slow because it is overloaded is pushed over the edge by clients that retry. Rules that hold: retry only on errors that are safe to retry (timeouts on reads, 503s), never on 4xx; cap attempts at two; use exponential backoff with full jitter so retries from thousands of servers do not line up; and apply a retry budget per partner, for example retries may be at most 10 percent of first attempts in any minute. The circuit breaker sits above this: when the failure rate over a 30-second window passes 50 percent with at least 20 samples, the circuit opens and calls fail immediately for 30 seconds, after which a handful of probe calls decide whether to close it. Failing fast is what lets the cache and the fallback do their job instead of every request burning its whole deadline.

How do we never charge twice when the payment call times out?

A timeout tells us nothing: the charge may or may not have happened. The only safe protocol is idempotency on the partner side: every charge carries a key derived from our order id, and the provider promises that a repeated key returns the original result instead of charging again. On timeout we do not re-send the charge; we call the provider's lookup-by-key endpoint, and if that also fails we park the order as payment unknown and let the deferred worker resolve it with backoff. The order state machine (pending, paid, failed, unknown) lives in our database, written before any outbound call, so a crash in the middle leaves a row that a reconciler can finish.

What do we show the user when parts are missing?

Decide per field. A shipping estimate can be stale or a range ("3 to 5 days") and say so. Address validation can be skipped and done asynchronously, with an email if it fails. Payment cannot be faked: if it is unknown at the deadline, the response is pending with a polling URL, and the UI says the order is being confirmed. Graceful degradation is a product decision expressed as a priority order over partner calls, and the aggregator enforces it with the deadline.

6. What a strong answer sounds like

• Lead with Little's law: in-flight calls equal rate times latency, which is why timeouts and bulkheads come before anything clever.
• Name the four tools in order: deadline-derived timeouts, idempotent retries with jitter and a budget, circuit breakers, and a stale-while-revalidate cache; say which one applies to reads and which to writes.
• Be explicit that a timed-out write is unknown, not failed, and that idempotency keys plus a status lookup are the only correct handling.
• Give the deadline number and show how it propagates into each call.
• Expected follow-ups: how to detect a partner that is slow for only some keys (per-shard breakers); how to test this (fault injection in staging); what metrics to alert on (partner p99, open circuits, deferred queue age).
• Common mistake: proposing a bigger thread pool or an async framework as the fix; concurrency is the symptom, not the cause.

02Where this question was reported

• Blind · Round 2; another commenter had a similar slow-APIs question