Notification System

What the interviewer is asking

Every product sends notifications. Orders says a package shipped, Security sends a sign-in code, Social says someone replied, Marketing announces a sale. Rather than each service dealing with Apple's and Google's push services, an SMS company and an email company itself, they all call one notification service.

The hard part is around the send: providers that limit you and sometimes fail, a sign-in code that must arrive in seconds while a million sale texts wait, and never sending a message twice. The walkthrough follows the 8 steps of the interview framework: scope, data, API, busy path, numbers, services, scale and deep dives.

1. Scope: what version 1 must do

• Send now, by push, text message (SMS) or email.
• Respect each user's settings: opt-outs per channel and per type, so Ana can turn off marketing texts and still get her sign-in codes.
• Templates: callers send a type and a few values; the wording ("Order {order} has shipped") lives in one place.
• No duplicates and retries: a failed send is tried again, and a retry never reaches the user twice.

Park the rest out loud: scheduled sends, A/B tests of message texts, analytics dashboards and an in-app inbox.

2. Data: notifications, settings and templates

record         fields
notification   id, user_id, type, channel, params, tries (time, provider, result)
settings       user_id, email, phone, push_tokens, opt_outs
template       type, channel, text

A notification is one message on one channel; its tries say what happened to it. The user's settings hold where to reach them (a push token is the address of one app on one phone, from Apple's or Google's push service) and what they turned off. A template exists per type and channel. Every send reads the settings by user id.

3. The API: one call, answered with 202

POST /notifications
{"user": "ana", "type": "order_shipped", "params": {"order": 81}, "idempotency_key": "k81"}
→ 202 Accepted   {"id": "n42"}

202 Accepted means "taken, not done yet". Sending can take seconds, or minutes when a provider is down, and no caller should wait for that: a 200 after delivery would let one slow provider hold up every service that sends notifications. The idempotency key (Idempotency Keys) makes a retried request safe: sent again with k81, it gets n42's answer and creates nothing new.

A provider's 200 only means it has the message. Whether an email bounced or a text was delivered comes later, by webhook:

POST /hooks/email   {"id": "n51", "status": "bounced"}
→ 200 OK

Check the signature, queue the event, answer, then update the notification (Webhooks). A bounce marks the address bad, since mail that keeps bouncing hurts your sender reputation. Push services answer at once instead: they reject a token that's no longer valid, and you delete it.

4. The busy path: accept fast, send later

1. Orders sends POST /notifications with an idempotency key.
2. The service checks the key, then reads Ana's settings: push is on.
3. It fills the template, saves n42 and answers 202.
4. In the background, it sends n42 to Apple's push service.
5. Apple's service delivers it to Ana's iPhone.

Orders is done at step 3, a few milliseconds in. The slow part comes after, at the providers: Apple's and Google's push services (how you reach an app that's closed, as in Real-Time Updates), an SMS gateway and an email provider. You don't control them, they limit you, and they fail now and then.

5. The numbers

From the interviewer: 100 million notifications a day, and a big event runs at 10× the average. The SMS company allows 1,000 texts a second and answers anything faster with 429 Too Many Requests. A day has 86,400 seconds; round it to 100,000.

quantity           working                                result
average            100 million ÷ 100,000 s                ≈ 1,000 a second (≈ 1,160 with 86,400)
peak               10 × 1,000                             ≈ 10,000 a second
API servers        10,000 ÷ 1,000 a server (assumed)      ≈ 10
a 1-million sale   1,000,000 ÷ 1,000 a second             1,000 s ≈ 17 minutes
its requests       1,000,000 ÷ 10,000 a second            100 s
waiting at 100 s   1,000,000 − 100 × 1,000                900,000
storage            100 million × 500 bytes                50 GB a day

The sale shapes the design. Its requests arrive within 100 seconds, but the gateway needs 1,000 seconds to send them however many servers you run, so 900,000 texts must wait: a queue's job (the Message Queues lesson). Storage is modest: 50 GB a day, about 1.5 TB for 30 days, 4.5 TB with 3 copies.

6. Services: an API, then a queue and workers per channel

With one app doing everything, sale texts waiting on the SMS gateway hold up pushes and codes too. Split accepting from sending:

• Notify API: checks the key, reads the settings, saves the notification, puts it on its channel's queue and answers 202. About 10 servers carry the peak.
• One queue per channel: push, SMS and email. With a stream, each channel is a topic and its workers one consumer group (the Pub/Sub and Streams lesson).
• Channel workers: take jobs from their queue, call the provider, record each try and retry failures.

Now a stuck SMS gateway backs up only the SMS queue.

7. Scale: the provider's limit, and urgent texts first

Say 4 SMS workers run, and each could send about 1,000 a second: together they'd try 4,000 a second and collect 429s. A quarter of the limit each, 250 a second, breaks when the count changes: 3 workers send only 750, 5 send 1,250. Instead, keep one shared token bucket per provider, 1,000 tokens a second in a fast in-memory store, and have every worker take a token before each send. It's the Rate Limiting lesson turned around: you're the client staying under someone else's limit.

The queue brings a new problem. At 10:01:40, 100 seconds into the sale, 100,000 texts have gone out and 900,000 wait. Ana asks for a sign-in code, valid for 10 minutes, and it joins the back of the queue. It leaves at 10:16:40, 15 minutes later: expired. More workers wouldn't help, since the gateway sets the pace.

The fix is a priority queue: urgent texts (codes, fraud alerts) get their own queue, and the workers always take from it first. Codes leave within a second, and the sale carries on right after. Keep a share of the limit for normal messages, so a flood of urgent ones can't starve them.

8. Deep dive: no duplicates

A worker waits up to 10 seconds for the gateway's answer, then counts the try as failed and retries. When the gateway is slow rather than broken:

time   what happens
0 s    try 1: the worker sends n77, Ana's sign-in code
2 s    the gateway texts Ana, and her phone shows the code
10 s   no answer yet: the worker counts try 1 as failed
11 s   try 2: a new request to the gateway, which texts Ana again
13 s   try 1's answer, "sent", arrives too late

Ana gets two codes. A sent-log doesn't help here: it marks n77 sent only once the gateway answers, and that answer came late. Never retrying after a timeout trades duplicates for lost codes. The fix is to send the notification id as an idempotency key with every try, so the gateway drops a repeat, wherever a provider takes one. Where none does, nobody can tell whether a timed-out try went out: delivery is at least once, and for a code a rare second text beats none.

Repeats start in three places, each with its own guard:

• The caller retries, or two copies arrive at once: the idempotency key at your API, claimed before anything is queued.
• The queue hands a job out twice, after a worker sent it and crashed before acking: a sent-log of ids, checked before each send.
• The provider's answer is lost: only the provider can catch it, by the id you send. For pushes, use the notification id as the collapse id (Apple's apns-collapse-id, a notification tag on Android): copies with the same one show as a single notification.

9. Deep dive: provider outages

At 10:00 the SMS gateway starts failing every request with 503; it's back at 10:10. Texts that can wait, like Ben's "arriving today" text n90, are retried with backoff: waits that double, 1, 2, 4 and 8 minutes, each with a few random seconds of jitter so texts that failed together don't retry together (the Timeouts, Retries, Circuit Breakers lesson).

try   time    result
1     10:00   503
2     10:01   503
3     10:03   503
4     10:07   503
5     10:15   delivered, 15 minutes late

If the gateway stays down for hours, try 5 fails too, and the message goes to a dead-letter queue and someone is alerted; once the gateway is back, dead letters are replayed or dropped as too old. Retrying forever would pile up texts that can never go, to numbers that don't exist.

Codes can't wait like that: one retried until 10:15 has expired. So sign up with a second SMS company and fail over when the circuit breaker on the main gateway opens. Say the backup costs more and allows only 200 a second: plenty for the few urgent texts, but the sale's 1 million texts alone would take 5,000 seconds there, about 83 minutes. Only urgent texts fail over; the rest wait for the main gateway.

10. Deep dive: per-user limits

Ana's photo takes off: 50 people like it within a minute, and Social asks for 50 pushes. Sending all 50 is how people end up turning notifications off, codes included; dropping all but the first hides the news.

time       push
10:00:02   Ben liked your photo
10:05:02   Cy and 48 others liked your photo

Instead, collapse them: a per-user, per-type limit, here one "likes" push every 5 minutes, sends the first at once and counts the rest into a summary. It's a rate limit whose extras are batched, not refused. Codes and order updates are never held.

What breaks first

• The database at the peak: 10,000 settings reads and 10,000 new records a second, about 4× what one server answers at 5,000 a second. Cache the settings, which rarely change, and split the records by user id.
• Settings that change while a message waits: during a sale a text can wait for minutes, so workers check opt-outs again just before sending.
• The token bucket's store: every send passes through it, so run it with a replica.

Summary: the order to follow

1. Scope: push, SMS and email now; opt-outs, templates, no duplicates, retries.
2. Data: notifications with their tries, user settings, templates.
3. API: POST /notifications with an idempotency key, answered 202; providers report back by webhook.
4. Busy path: check, read settings, save, queue, answer; the providers come after.
5. Numbers: about 1,000 a second, 10,000 at the peak; a million texts take 1,000 seconds.
6. Services: a Notify API, then a queue and workers per channel.
7. Scale: one token bucket per provider; urgent texts in their own queue, served first.
8. Deep dives: a duplicate guard at each hop; backoff with jitter, dead letters and a backup gateway for codes; per-user limits that collapse bursts.

The same pieces fit an email service or a webhooks platform: accept fast, queue per destination, respect its limits, make every retry safe.