Design cross-service transactions (sagas and 2PC)

01The design, in the six-step framework

1. Requirements

The prompt is abstract: "transactions across services that each have their own database", usually with a worked case such as place an order, charge the card, reserve the stock. Make the case concrete and design the protocol around it.

Functional

• An operation spanning several services (order, payment, inventory, shipping) either fully happens or is fully undone.
• Each service keeps its own database; no shared database and no global lock.
• A client can ask for the outcome of an operation and get one answer, even after retries and crashes.
• Operators can see in-flight operations and intervene when a step is stuck.
• Out of scope: the services' own logic, fraud checks.

Non-functional

• Scale: 5,000 orders a second at peak, 4 to 6 steps each, 20 to 30 thousand step calls a second.
• Latency: an order confirmed under 2 seconds at p99; compensation completes within a minute of a failure.
• Correctness: no money taken without an order and no order without stock; duplicates from retries never double-charge.
• Availability: a crashed coordinator loses no in-flight operation; a slow service delays only its own step.

2. Core entities

• Saga: saga id (the client's idempotency key), type, state (started, step k done, compensating, completed, failed), created at, version.
• Step: saga id, index, service, action request, compensation request, state (pending, done, failed, compensated), attempts.
• Outbox entry: per service, the message to send recorded in the same local transaction as the business write.
• Participant record: per service, (saga id, step) → result, for idempotent replies.

3. API

• POST /orders with an idempotency key; returns 202 with the saga id, or the final result if already known.
• GET /sagas/{id} returns state and per-step status; the client polls or receives a push.
• Participant contract, each service: POST /{resource}/reserve, /confirm, /cancel with the saga id and step index; every call idempotent on (saga id, step).
• Operator: GET /sagas?state=stuck, POST /sagas/{id}/retry-step, POST /sagas/{id}/compensate.

4. High-level design

1. The client posts the order with an idempotency key; the orchestrator writes a saga record (state started) to its replicated log and returns 202.
2. It runs steps in order: create the order (pending), reserve inventory, charge payment, confirm the order. Each request carries the saga id and step index; each participant performs a local transaction and writes its reply to its outbox in the same transaction.
3. Outbox relays publish replies on the bus; the orchestrator advances the saga, recording each step's result in the log before issuing the next.
4. If a step fails permanently, the orchestrator runs compensations in reverse (release stock, refund, cancel order), each also idempotent, and marks the saga failed with a reason the client can read.

5. Deep dives

2PC or sagas, and when is each right?

Two-phase commit gives atomic visibility: no one sees the order without the charge. It needs every participant to hold locks from prepare to commit and a coordinator whose log decides the outcome; a slow or dead participant blocks the others, and a dead coordinator leaves participants in doubt until it returns. That is acceptable inside a database (which is where 2PC lives in practice) and wrong across autonomous services at 5,000 a second. Sagas give atomicity in the end: each step commits locally and a failure is undone by compensations, so there is a window where the order exists and the charge does not. Design for that window: the order is visible as pending, stock is reserved not sold, and the card is authorised not captured until the saga completes. Say the rule: 2PC when intermediate states must be invisible and participants are few and fast; sagas otherwise (distributed transactions).

How are retries and duplicates made harmless?

Every call carries (saga id, step index) as its idempotency key. A participant records the key with its result in the same local transaction as the business write, so a retried request finds the record and returns the stored result without acting again; a crash between the write and the reply is covered because the reply lives in the outbox and the relay resends it. The orchestrator's own log makes it restartable: a worker that crashes mid-saga is replaced by another that reads the log and resumes from the last recorded step. Without the recorded result, a retried "charge" could take money twice; this is the single most probed point.

What are the compensation rules, and what cannot be compensated?

Compensations are not rollbacks; they are new business actions (refund, release) that may themselves fail and must be retried until they succeed, which is why each is idempotent and why the saga log keeps retrying with backoff and alerts after a threshold. Some steps are not compensatable (an email already sent, a shipment already picked up); order the saga so such pivot steps come last, after everything that can fail has succeeded. Concurrency between sagas is the other subtlety: two sagas reserving the last unit race at the inventory service, which is fine because the reservation is a local transaction there; a saga reading a value another saga later compensates is a dirty read and must be tolerated by design (reservations expire, counts are rechecked at confirm).

Orchestration or choreography, and how do timeouts and ordering work?

Choreography (each service reacts to the previous service's event) has no central log and is harder to observe and to reason about when steps are added; orchestration (a coordinator drives steps from a log) is explicit and gives operators a place to look. Choose orchestration for flows with more than three steps or with compensations. Timeouts are per step and tuned to the participant (payment 2 seconds, inventory 300 ms); a timeout is not a failure, it is unknown, so the orchestrator queries the participant's record for (saga id, step) before deciding to retry or compensate. Ordering inside one saga comes from the log; across sagas there is none, by design.

6. What a strong answer sounds like

• Make the example concrete, then state why 2PC across services is the wrong tool at this scale and keep it for atomic-visibility cases.
• Define the saga, its log, and the three-way idempotency (key, stored result, outbox) that makes retries safe.
• Explain compensations as forward business actions, retried until done, with pivot steps last.
• Choose orchestration and say what operators can see and do.
• Follow-ups to expect: reads during the window (expose pending states), exactly-once across the bus (at-least-once plus idempotent consumers), and how a new step is added (a new saga version, old in-flight sagas finish on the old definition).

02Where this question was reported

• PracHub · 'distributed transactions protocol', Hard
• interviewkickstart.com · Q14: distributed transactions across services each with its own database