Idempotency and exactly-once

Making a repeated request harmless, which is how 'exactly once' is really built.

System design conceptQueues, streams and batch1 Nov 2026

01The intuition

The problem it solves

You pay for a train ticket on your phone. You press Pay, the request leaves the phone, the bank server charges your card, and on the way back the reply is lost in a tunnel. Your phone sees nothing and shows "retry". You press Pay again. The server, which has no memory of the first request, charges you again. Two charges, one ticket.

Every network loses replies. Every sensible client retries. So every server will eventually receive the same request twice, and the only question is whether receiving it twice does the thing twice. An operation that can be applied many times with the same effect as once is called idempotent. Setting a light switch to "on" is idempotent; pressing a toggle is not. "Set the balance to 500" is idempotent; "subtract 100" is not. The whole art is turning the second kind into the first.

Build it one piece at a time

Step 1: name every request. Before sending, the phone generates a random idempotency key, say k7f2…, and attaches it to the request. A retry reuses the same key. Two taps by the user are two different keys, because the user really wants two tickets then.

Step 2: remember what you did. The server keeps a table keyed by idempotency key: the key, the result it produced, and when. On arrival it checks the table.

Arrival	Key in table?	Action	Charges made
Pay, key k7f2	no	charge card, store k7f2 → ticket 991	1
retry, key k7f2	yes	return ticket 991 again, charge nothing	1
new tap, key p3a9	no	charge card, store p3a9 → ticket 992	2

Step 3: close the race. Two copies of the same request can arrive at two servers in the same millisecond, both see "not in table", and both charge. The check and the insert must be one atomic step: insert the key first with a unique constraint, and only the request whose insert succeeded does the work. The loser waits and returns the winner's result.

Step 4: make the operation itself idempotent where you can. Instead of "add 1 to the like count", store "user 17 likes video 42" as a row with a unique key; inserting it twice is a no-op. Instead of "append this event", write it under the event's own ID. A database upsert and a set-to-value are idempotent by nature; increments and appends are not, so wrap them.

Step 5: expire the memory. The key table cannot grow forever. Keep keys for as long as a client might retry, typically 24 hours, then delete them. A retry after that would be treated as new, which is why clients stop retrying long before.

Why it works, with the numbers

The invariant is: for each idempotency key, the side effect happens at most once, and every delivery of that key returns the same result. Combined with a queue or a client that delivers at least once, you get the effect people call exactly-once:

at-least-once delivery + idempotent processing = exactly-once effect

The cost is one key lookup per request and one row per request for a day. At 10,000 payments a second that is 864 million rows a day, about 50 GB at 60 bytes a row, which a key-value store holds easily. The failure modes: a client that forgets to reuse the key (two charges again), a key table that is not atomic (the race above), and a result that is too big to store, in which case you store a pointer to it.

The trade-off you must say out loud

Idempotency adds a lookup, a table and a rule every client must follow. Say where the key is generated (the client, so a retry carries it) and how long it lives. Say which operations are naturally idempotent and which need the table. The follow-up is "what about exactly-once in the queue itself", and the answer is that the queue gives you at-least-once and your consumer's dedup gives you the rest; there is no magic in the middle.

Where it shows up in the 100 Google problems

Google Pay: a transfer is keyed, inserted atomically, and never applied twice however many times it is sent.
Ticketmaster: confirming a held seat twice returns the same booking, not two.
Notification system: a redelivered message must not buzz the phone twice.
Ad click aggregation: duplicate click events carry the same ID and are counted once in billing.
Managed pub/sub: the consumer side of at-least-once delivery is this table.