Webhooks

The problem: has the payment gone through?

Your shop takes payments through a payment provider. Ana orders a lamp and pays $49 on the provider's page, so the payment succeeds on the provider's servers, not yours. Your shop ships order 81 only once it's paid, so it has to find out.

The first idea is polling: ask the provider about every pending payment every 5 seconds. With 1,000 payments pending, that's 200 requests a second. A payment that takes about a minute gets about 11 "still pending" answers for every useful one, and your shop still learns up to 5 seconds late. Trusting Ana's browser to report back from your thank-you page is worse: she may close the tab, and anyone can open that page.

How a webhook works

A webhook turns it around: you give the provider a URL, and it sends a request there the moment something happens. It's push, like a WebSocket, but between two servers, so your side needs only an HTTP endpoint:

POST https://shop.com/hooks/payments
Signature: t=10:00:00, hmac=c466fc16…

{"id": "evt_1",
 "type": "payment.succeeded",
 "order": 81}

The body is an event: an id, a type and what it's about. A 2xx answer says "got it"; any other answer, or none, is a failure.

Answer fast, work later

The provider waits only so long for your answer: say 10 seconds. If your endpoint does all the work first (mark the order paid, buy a shipping label from the courier's API, email Ana), 12 seconds is easy to reach, and the provider counts the event as failed even though you handled it.

So the endpoint checks the request, puts the event on a queue and answers 200 at once; an order worker does the slow part. Save the event before you answer: answer first and crash, and the provider won't send it again.

Retries, and the same event twice

When no 2xx comes back, the provider tries again with growing waits, the backoff from message queues. Say its plan is this one:

try   wait before it   time
1     none             10:00
2     1 minute         10:01
3     5 minutes        10:06
4     30 minutes       10:36
5     2 hours          12:36
6     6 hours          18:36
7     12 hours         6:36 the next day

Say your server is down from 10:00 to 10:20. Tries 1 to 3 fail; try 4, at 10:36, gets through, 16 minutes after you're back: order 81 ships 36 minutes late, but it isn't lost. Try 7 comes about 21 hours after the first; then the provider gives up.

Retries bring duplicates. If try 4 reaches you but its 200 is lost on the way back, the provider sends evt_1 again at 12:36: it can't tell a lost answer from a lost request. So webhooks are at-least-once, like the jobs in a queue. The event keeps its id on every try: keep the ids of events you've handled, and skip one you've seen. The id works like an idempotency key (see Idempotency Keys): the sender makes it, the receiver checks it. Write the id in the same database transaction as the order change, so a crash can't leave one without the other.

Signatures: is it really the provider?

Your webhook URL is public: anyone can post {"type": "payment.succeeded", "order": 99} to it and get an order shipped that nobody paid for. So the provider signs every request with a secret only it and your shop know: it computes an HMAC (with SHA-256) of the time t, a dot and the raw body, and sends the result in a header, never the secret.

Your endpoint computes the same HMAC over what it received and compares. With the toy secret toy-secret-shop-81:

body        HMAC starts   header says   result
order 81    c466fc16      c466fc16      match: accept
order 99    a54cdd07      c466fc16      no match: reject

Change one character and the HMAC changes completely, and nobody can compute the new one without the secret. Compute it over the raw bytes, before parsing the JSON, and compare in constant time:

import hashlib, hmac

def is_real(secret, t, body, header_hmac):
    expected = hmac.new(secret, t + b"." + body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, header_hmac)

A signature alone doesn't stop a replay: a real request copied from a log still matches a day later, since the same secret and bytes always give the same HMAC. That's why t is signed too: your endpoint rejects any t more than about 5 minutes old. The provider signs each try afresh, so its own retries pass, and a copy sent within the 5 minutes is caught by the event ids you keep.

Why not something simpler? An IP allow-list says where a request came from, not what's in it, and the provider's addresses change; it's a fine extra layer on top. A secret in the URL travels with every request into server and proxy logs, where whoever reads it can send any event. HTTPS protects a request on its way, but doesn't stop a stored copy from being sent again.

Events out of order

Retries also reorder events. Say your server restarts from 10:00 to 10:02, just as Ana pays, and at 10:03 she cancels and is refunded:

time    what reaches you             order 81, as written
10:00   evt_1 succeeded: no answer   awaiting payment
10:01   evt_1 succeeded: no answer   awaiting payment
10:03   evt_2 refunded: 200          refunded
10:06   evt_1 succeeded: 200         paid, shipped

At 10:06, evt_1 isn't a duplicate (it was never handled) and its t is fresh, so the worker ships a refunded order. Waiting a minute per event, or comparing arrival times, doesn't help: arrival order is the problem. Two fixes work:

• Versions: each event carries the payment's state and a version (or the time it happened). Apply it only if it's newer than what you have: evt_1 brings version 1, order 81 already has version 2, so evt_1 is skipped.
• Fetch the state: treat each event as a nudge and ask the provider's API for the payment's current state. It costs a request per event, but it's always up to date.

Sending webhooks

When you're the provider, each delivery is slow work on a server you don't control, so it goes through a queue: delivery workers send it and retry each endpoint on its own schedule, so one customer whose server is down doesn't hold up the others. After the last try, the delivery goes to a dead-letter list, and a dashboard shows customers every try and its answer, with a button to send an event again.

Where it shows up in design interviews

Any design that takes payments or works with a partner has webhooks coming in: checkout, subscriptions, a courier's tracking updates. When the interviewer asks how your system learns a payment went through, walk one event through: check the signature and the time, queue the event, answer 200; the worker skips handled ids and applies only newer states. Verified, queued, deduped, order-safe.

Common interview questions

Why use webhooks instead of polling?

Polling sends requests whether or not anything changed, and learns of a change up to one interval late. A webhook arrives when the event happens. Many systems still poll now and then as a safety net, for an event that never arrived.

How do you handle the same event arriving twice?

Expect it: delivery is at-least-once. Keep the ids of handled events, written in the same transaction as the change they made, and skip an event whose id you've seen.

How do you know a webhook really came from the provider?

Check its signature: an HMAC of the timestamp and the raw body with a shared secret, compared in constant time. Reject old timestamps to stop replays.

What if events arrive out of order?

Never trust arrival order. Apply an event only if its version or time is newer than what you've stored, or fetch the current state from the provider's API when an event arrives.

Summary

• A webhook is the provider calling your URL when an event happens: push between servers, instead of polling.
• Answer fast: queue the event, answer 200 inside the timeout, and let a worker do the work.
• Retries with backoff make delivery at-least-once: skip event ids you've handled.
• Check the HMAC signature over the time and the raw body, and reject old timestamps.
• Events can arrive out of order: apply only newer versions, or fetch the current state.
• As the sender: a delivery queue, retries per endpoint, dead letters and replay.