Payment System

What the interviewer is asking

"Design a payment system" usually means: a marketplace takes card payments through an outside payment provider (it never talks to the card networks itself), refunds them and pays its sellers. The running example: Ana buys a $40 lamp from Bo, one of the sellers.

The traffic is small; the interviewer listens for correctness: no double charges, no lost payments, books that always add up. The walkthrough follows the 8 steps of the interview framework: scope, data, API, busy path, numbers, services, scale and deep dives.

1. Scope: what version 1 must do

• Charge a card for an order.
• Refund all or part of a payment.
• Pay sellers their share later, in a nightly payout.
• Never charge twice, never lose a payment.
• An audit trail: a record of every money movement.

Park your own fraud scoring, other currencies and subscriptions. And say that you never store card numbers: the provider's payment form takes the card and gives you a token to charge it with. Keeping card numbers would bring your systems under PCI DSS, the card industry's security rules, in full, and nobody may keep the security code once a payment is approved.

2. Data: payments and a double-entry ledger

Every payment gets a record:

field             example
id                p81
order_id          o81
amount            4000 (cents)
status            created → pending → succeeded / failed; succeeded → refunded
provider_ref      ch_9
idempotency_key   k81

The status only moves forward. The provider's reference matches its webhooks and reports to p81. The money goes in a double-entry ledger: every movement is two entries, out of one account and into another, that sum to zero (accountants call the two sides a debit and a credit). A balance is the sum of an account's entries:

entry   movement   account   amount
1       charge     Ana       −$40
2       charge     Bo        +$40

The ledger is append-only: rows are added, never updated or deleted. That makes it the audit trail, with a check built in: if all entries don't sum to zero, there's a bug. A balance column per seller would lose the history with every update.

3. The API: three calls

POST /payments  {"order_id": "o81", "amount": 4000, "idempotency_key": "k81"}
→ 201 Created   {"id": "p81", "status": "succeeded"}

POST /refunds   {"payment_id": "p81", "amount": 1000, "idempotency_key": "r81"}
→ 201 Created   {"id": "rf1"}

POST /hooks/provider   (the provider calls this one)
{"event": "e7", "payment": "p81", "status": "succeeded"}
→ 200 OK

Amounts are whole cents in integers. Floating-point numbers can't hold most cents exactly (0.10 + 0.20 gives 0.30000000000000004), and payment APIs commonly take amounts in the currency's smallest unit. The idempotency key (the Idempotency Keys lesson) makes a retry safe: a retry with k81 gets the same p81 back, as a 200 or as the first reply replayed, and nothing is charged twice.

Some payments finish later, say when Ana's bank asks her to confirm in its app: the provider answers pending and reports the result by webhook. As in the Webhooks lesson, the handler checks the signature, queues the event and answers 200; a worker updates the payment, skipping event ids it has handled.

4. The busy path: save, charge, record

1. Checkout sends POST /payments for o81 with key k81.
2. The payment service saves p81 as created, with the key.
3. It marks p81 pending and asks the provider to charge $40, sending p81 as the provider's idempotency key.
4. The provider charges Ana's card and answers: succeeded.
5. In one transaction it saves succeeded and the ledger entries, then answers 201.

Saving before calling means a crash halfway leaves a record to check: no payment is lost. The status and its entries go in one transaction, both or neither (the Transactions and Isolation lesson). The risky step is the provider call, someone else's system over a network: a slow or failed call is this design's main risk.

5. The numbers

From the interviewer: 1 million orders a day, one payment each, and a big sale day at 10× the average. A day has 86,400 seconds; round it to 100,000.

quantity         working                           result
average          1 million ÷ 100,000 s             ≈ 10 a second (≈ 12 with 86,400)
peak             10 × 10 a second                  ≈ 100 a second
ledger entries   1 million × 4 (charge and fee)    4 million a day
ledger size      4 million × 500 bytes             2 GB a day, about 730 GB a year

One database server answers about 5,000 simple lookups a second, and 730 GB a year (about 2.2 TB with 3 copies) fits one database with replicas for years. Correctness, not scale, drives this design. One database also lets one transaction write a payment and its entries; sharding by account would split a movement's two entries across servers.

6. Services

• Payment service: the only part that talks to the provider and writes the ledger.
• Webhook handler and event queue: verify, queue, answer 200; workers apply each event to its payment.
• Nightly jobs: payouts and reconciliation.

For payouts, a nightly job pays each seller their ledger balance, one payout each. One transfer per seller per day costs far less than one per sale, and fees and refunds are already taken off. A payout is two ledger entries, sent with an idempotency key made of the seller and the day, so a rerun pays nobody twice. Many marketplaces also wait a few days before paying out, so refunds and disputes can still come out of the balance.

7. Scale: two writers, one payment

Several payment servers and event workers mean one crash stops nothing, but they race. The provider sends its webhook even for payments it answered at once, so p81's answer and its webhook can arrive together. Both read p81 as pending and both write the charge: Ana's balance says −$80 for a $40 lamp, and the ledger still sums to zero.

The fix, from the Concurrency and Locks lesson, is one conditional update in the same transaction as the entries:

UPDATE payments SET status = 'succeeded'
WHERE id = 'p81' AND status = 'pending';
-- one row changed: write the entries; none changed: stop

The database checks and writes in one step, so exactly one writer wins. Skipping handled event ids doesn't help: one update came from the answer, one from the webhook. The same rule keeps statuses moving forward, so a late webhook can't turn a refunded payment back into succeeded.

8. Deep dive: the uncertain call

Say the payment service waits up to 10 seconds for the provider, then gives up (the Timeouts, Retries, Circuit Breakers lesson). One sale-day morning:

time   what happens
0 s    p81 is pending; the charge for $40 goes out with key p81
1 s    the provider charges Ana's card; its answer is lost on the way back
10 s   the timeout ends the wait; p81 is still pending
11 s   a retry with the same key p81: the provider returns the first result

A timeout means unknown, not failed: the request may have been lost on the way there (nothing charged) or the answer on the way back (charged, as here). Don't guess: retry with the same key, so the provider returns the first result, or ask it for p81's status. A new key looks like a new payment and charges Ana again; marking p81 failed either has her pay twice or keeps her $40 with no lamp. If the provider stays unreachable, keep the payment pending, tell Ana you're confirming it, and keep checking in the background.

9. Deep dive: reconciliation

Webhooks get lost. Each night the provider sends a settlement report of every charge and refund it made, and a job compares it with your payments: succeeded must appear as charged, refunded as refunded, and failed not at all, since no money moved.

payment   your records      provider's report
p81       succeeded · $40   charged · $40
p82       succeeded · $25   charged · $25
p83       failed · $60      charged · $60      mismatch
p84       refunded · $15    refunded · $15
p85       failed · $30      (not in the report)
p86       succeeded · $90   charged · $90

p83 says failed, but the provider charged $60: say it was given up during a long outage, and the webhook that said it succeeded was lost. Fix it from the report: mark p83 succeeded, write its entries, then ship the order or refund it. Alert someone when the night's mismatch count isn't zero (the Monitoring and SLOs lesson). Webhooks bring news in seconds; reconciliation catches what they lose.

10. Deep dive: the double-entry ledger

Ana pays $40, the marketplace keeps a 10% fee, and Bo gives her $10 back for a scratch on the shade:

entry   movement   account    amount
1       charge     Ana        −$40
2       charge     Bo         +$40
3       fee        Bo         −$4
4       fee        our fees   +$4
5       refund     Bo         −$10
6       refund     Ana        +$10

Bo has 40 − 4 − 10 = $26, Ana −$30, your fees +$4, and all entries sum to $0. Now say Bo's plan charges 5%, so his fee should have been $2. Don't edit rows 3 and 4: add a correcting movement, $2 out of your fees into Bo's account. His balance becomes $28, and the record keeps the first fee and the fix. A single +$2 entry would break the zero sum: money from nowhere.

11. Deep dive: a saga across services

An order takes three services, each with its own database: orders creates o81, payments charges $40, stock reserves the lamp. If stock fails after the charge, there's nothing to roll back: each service has committed, and the charge happened at the provider. A saga gives each step an undo, a compensating step, run newest first:

step   action         result
1      create order   done
2      charge $40     done
3      reserve lamp   out of stock
4      refund $40     done (key r81)
5      cancel order   done

The refund has its own idempotency key, so a retry refunds once. Many shops reserve stock before charging, so the undo is releasing a hold instead of a refund.

What breaks first

• The provider on a sale day: timeouts, retries with the same key, a circuit breaker, payments left pending.
• Webhooks: lost or late; statuses only move forward, and reconciliation catches the rest.
• The database: one leader holds every payment; keep a replica in another zone.
• Payouts: a crash halfway; the payout key makes a rerun pay each seller once.

Summary: the order to follow

1. Scope: charge, refund, nightly payouts; never twice, never lost; an audit trail; no card numbers.
2. Data: payments with a status and a key; an append-only double-entry ledger.
3. API: POST /payments and POST /refunds with idempotency keys and amounts in cents; a webhook from the provider.
4. Busy path: save first, call the provider, then the status and entries in one transaction.
5. Numbers: about 100 payments a second at the peak: correctness, not scale.
6. Services: one payment service, a webhook handler with a queue, nightly jobs.
7. Scale: one conditional update, so a status moves once.
8. Deep dives: a timeout is unknown, so retry with the same key; reconcile each night; the ledger sums to zero and is fixed with new entries; a saga undoes with a refund.

The same pieces fit a digital wallet or subscription billing: a key on every call that moves money, a ledger that sums to zero, a nightly check.