Idempotency Keys

The problem: a retry that charges twice

A shop's app sends POST /payments with {"amount": 30}. The payment service asks the card network to charge Ana's card $30, saves the payment as p7 and sends back 201 Created. On a weak phone signal, that reply never arrives. After 10 seconds the app gives up waiting (a timeout) and sends the same request again.

The timeout happened only in the app: the server finished its work and never heard that the app stopped waiting. To the server, the second request looks exactly like the first, with the same path, amount and card. It can't tell a retry from a second purchase, so it charges again and saves p8. Ana pays $60 for a $30 order.

Not retrying is no answer either. The app can't know whether the payment went through, so Ana presses Pay again (the same second request) or walks away from an order she has paid for. Retries are needed; they have to be safe.

How idempotency keys work

A request is idempotent when sending it twice has the same effect as sending it once. GET, PUT and DELETE are idempotent by design; a POST that creates something is not. An idempotency key makes such a POST idempotent:

1. The client makes a unique key for each payment, a random id such as a UUID, before its first try.
2. It sends the key with every try of that payment, usually in an Idempotency-Key header. That's a common convention, supported by payment APIs such as Stripe's, and an IETF draft describes it as a standard header.
3. The server keeps a record per key: the request it came with and the reply it sent.
4. A request with a new key is new work. A request with a key the server has seen gets the stored reply, and nothing is charged again.

POST /payments
Idempotency-Key: 4f9c2a7e-8b1d-4c3a-9e6f-2d5b7a1c0e93
{"amount": 30}

→ 201 Created
{"payment": "p7"}

Here is a run, with keys written k1 and k2 for short:

request            key store has     the API does             charged
POST $30, key k1   no k1             charges, saves p7        $30
POST $30, key k1   k1: $30 → p7      sends p7's reply again   $30
POST $30, key k2   no k2             charges, saves p8        $60
POST $45, key k1   k1: $30 → p7      422 error, key reused    $60

The first reply is lost; the retry gets the reply it carried, and the card is charged once. Ana's second item has its own key, so it's a new payment. The last line is a bug in the app, covered below.

The hard parts

Two copies at the same moment. A double tap on Pay, or a retry sent while the first try is still running, can put two requests with the same key on two servers at once. If each server looks the key up first and stores it only after charging, both find nothing and both charge: the key alone doesn't stop the race. The fix is to claim the key first: insert it with the status "in progress" into a table where the key is unique, and charge only if that insert worked.

moment   Server A                       Server B
1        inserts k1 "in progress": ok
2                                       inserts k1: fails, key taken
3        charges $30, saves p7          answers 409 Conflict
4        stores k1: 201, p7, done

The second insert fails, so that server charges nothing and answers 409 Conflict ("still in progress"), or waits and returns the first reply. Checking and claiming are now one step.

The same key with a different body. Store a fingerprint of each request (a hash of its body) with its key. If a request arrives with a known key but a different body, say k1 again but for $45, the client has a bug. Return an error (the IETF draft suggests 422) rather than the old reply: sending back p7 would tell the app a $45 order is paid when only $30 was charged.

Where the key comes from. The client, not the server. Only the client knows which tries belong to one payment, and it has the key before anything can be lost. A key the server makes and returns in its reply is lost along with that reply. A key built from the payment's contents, such as the card and the amount, is no better: two real $30 purchases would share it, and the second would never be charged. A UUID has 122 random bits, so two clients practically never make the same one.

How long to keep keys. Long enough to outlive every retry, then delete them. Say 1 million payments a day and about 500 bytes per key record: kept 24 hours, the store holds about 500 MB and stays that size; kept forever, it grows by 500 MB a day, about 180 GB a year, for keys nobody will send again. Stripe, for example, may remove keys once they're at least 24 hours old. Too short is worse than too long, since a retry after its key expired is a new payment, so the client must stop retrying well before then.

Failures. A declined card is a final answer: store that reply too, so a retry gets the same decline. If a server crashes after claiming a key but before it knows the outcome, the key is stuck "in progress" until the server settles it, for example by asking the card network whether the charge happened.

Which calls need a key

Calls that create something or move money: creating a payment, sending a transfer, creating an order, placing a bid. Each send does the work again. Calls that set a value (a profile name, an order's status), delete something or read leave the same state however often they're repeated, so they're safe as they are. In a REST API, that's mostly the POSTs.

Where it shows up in design interviews

Any design where money moves or a retry could act twice: a payment system, a digital wallet, checkout, ticket booking, a flash sale, and any worker taking jobs from a queue, where delivery is at-least-once and the job id can serve as the key. Say "an idempotency key on every create call", then how it's stored: a table with the key unique, the request's hash, a status (in progress or done) and the stored reply, kept for about 24 hours. Then name the race and the claim-first insert.

Common interview questions

What is an idempotency key, and who makes it?

A unique id for one attempt at an operation, sent with every retry of it, so the server can answer a repeat with the result it already has. The client makes it before the first try: a key made by the server would come back in the reply, and the reply is exactly what can get lost.

What if two requests with the same key arrive at once?

Without care, both look the key up, find nothing and both charge. Claim the key first with an insert under a unique constraint: only one insert succeeds, and the other request gets 409 Conflict or waits for the first one's reply.

Why not block repeated charges of the same amount on the same card for a minute?

Two real purchases can look exactly alike, so a real second one is refused, and a phone that was offline can retry later than any window you choose. Only the client knows whether a request is a retry, and the key is how it says so.

How long should keys be kept?

A fixed time well past the longest retry, often 24 hours. Forever wastes storage that grows every day; too short turns a late retry into a second charge.

Summary

• A lost reply and a retry make a POST do its work twice: $60 for a $30 order.
• The client makes an idempotency key per payment and sends it with every try; the server stores key → reply and answers repeats from it.
• Claim the key first, as "in progress" under a unique constraint, so two copies at once can't both charge.
• A known key with a different body gets an error, not the old reply.
• Keep keys a fixed time, such as 24 hours, and put them on calls that create something or move money.