Webhooks Service

The problem

"Design the webhooks service of a payments platform." Customers register a URL and the events they want, such as payment.refunded, and your service POSTs each event to their servers. There are about 50,000 endpoints and you control none of them: some answer in a tenth of a second, some take seconds, some are down for days.

This practice problem reuses the pattern of Design a Notification System, with customers' servers in the place of the SMS gateway. What's new comes from those endpoints: isolating a dead one, giving up on one, signatures, order per payment, and a log to search and resend from.

The numbers

From the interviewer: 100 million deliveries a day, and a healthy endpoint answers in about 0.1 s. You wait up to 10 seconds for an answer. Round a day to 100,000 seconds (86,400 really).

quantity          working                      result
deliveries        100 million ÷ 100,000 s      ≈ 1,000 a second (≈ 1,160 with 86,400)
waiting at once   1,000 × 0.1 s                ≈ 100

So about 100 workers, each sending one at a time, do the whole job, until an endpoint makes them wait 10 seconds.

A dead endpoint: isolate each one

Say 200 workers share one queue, and Shop B, which gets 1 in 10 of all deliveries, stops answering. Each try to B holds a worker for 10 seconds, so B alone would hold 100 a second × 10 s = 1,000 workers. Within 5 seconds nearly all 200 wait on B; a minute in, everyone else's deliveries are about 50 seconds late, and later every second.

More workers don't fix it: B wants five times the pool. A 1-second timeout still lets B hold 100 workers, more than all other customers need together (900 a second × 0.1 s = 90), and fails every healthy endpoint that takes 2 seconds. Instead, isolate each endpoint:

• A queue per endpoint: cheap lists keyed by endpoint id, drained by one shared pool of workers.
• A concurrency limit per endpoint, say 10 at once: B holds 10 of the 200 workers.
• A circuit breaker per endpoint: while it keeps failing, pause its queue and try one delivery now and then.

Giving up: turn it off, tell the owner

Each failed delivery is retried with backoff and jitter, as in the Webhooks lesson: waits of 1, 5 and 30 minutes, then 2, 6 and 12 hours. That's 7 tries over 1,236 minutes, about 21 hours, before it's marked failed.

At 10 at once and a 10-second timeout, B's queue sends 1 try a second while 100 new deliveries arrive: it grows by about 10 million a day. Retrying forever tells nobody; deleting loses the events. A common rule: email the owner when failures start, and after a few days of nothing but failures turn the endpoint off and email again, linking the failed deliveries. Missed events stay in the log, say for 30 days, to resend later.

Signing every delivery

POST https://shop-a.com/hooks
Webhook-Id: evt_3
Signature: t=10:03:00, hmac=704665cc…

{"type": "payment.refunded", "payment": "p_81", "version": 2}

As in Webhooks, an HMAC (SHA-256) of the time t and the body lets the receiver check a delivery is yours, unchanged and recent. The sender's decision is the secret: one per endpoint, made when the customer adds it and shown only to them. One secret for all customers would let any of them forge events for another. HTTPS proves to you which server you reached, not to the receiver who is calling. To change a secret, sign with both the old and the new one for a while, say a day.

Order per payment

Shop A wants each payment's events in the sequence they happened. At 10:05, evt_1 has failed twice and waits for its next try at 10:06:

made    event   payment   type        what the queue does
10:00   evt_1   p_81      succeeded   retries at 10:06
10:02   evt_2   p_82      succeeded   sent
10:03   evt_3   p_81      refunded    waits for evt_1
10:04   evt_4   p_83      succeeded   sent

Sending evt_3 now lets the refund overtake its payment. Holding all of A's events behind evt_1 is head-of-line blocking: p_82 and p_83 would wait for nothing. Keep one line per key, the payment id, inside each endpoint's queue, so a stuck event holds up only its own payment. Events still carry a version for receivers to check.

The delivery log and resending

Every try is a row: event id, endpoint, payment id, time, status code and the start of the answer. Key it by endpoint and time, with indexes on event id and payment id, so a search reads one customer's rows. At 100 million tries a day of about 500 bytes, that's 50 GB a day, 1.5 TB for 30 days, 4.5 TB with 3 copies.

Shop A never got yesterday's refund for p_95. The log shows evt_7's 7 tries, from 10:00 yesterday to 06:36 today, all answered 500 by a bug A has since fixed. Resend sends evt_7 again, same id, so A can skip a copy it already has; signed afresh, since a day-old t fails A's 5-minute check; and through A's queue, keeping p_95's sequence and A's limit. A new id could make A refund twice.

Same pattern as the Notification System

An API saves each event, answers 202 and queues one delivery per subscribed endpoint; workers send, record each try and retry with backoff; every send carries an id, since delivery is at least once. Endpoints replace the providers, and the per-endpoint limit the provider's token bucket.

Summary

• Numbers: about 1,000 deliveries a second, 100 waiting at once.
• Isolation: a queue per endpoint, 10 at once, paused while it fails.
• Giving up: 7 tries over about 21 hours; after days of failures, off and the owner emailed.
• Signatures: an HMAC of t and the body, a secret per endpoint.
• Ordering: one line per payment inside the endpoint's queue.
• Resending: the same event id, signed afresh, through the queue.