Pub/Sub and Streams

The problem: one event, many services

When a customer places an order, several services need to know: Email sends the receipt, Stock sets the items aside, Reports counts the sale, a fraud check looks at the payment. Each needs every order.

A message queue doesn't do this. It gives each message to one reader and deletes it once handled, which is right when several workers share one job. Point four services at one queue and they share it the same way: each gets about a quarter of the orders.

The quick fixes fall short. A queue per service means Orders must know every reader and change for each new one. Calling each service's API makes the customer wait for the slowest, and a service that's down misses orders.

Pub/sub: publish once, every subscriber gets a copy

In publish/subscribe (pub/sub), the producer publishes each event once to a topic, and every service that subscribes to the topic gets its own copy: a fan-out. Orders doesn't know who reads, so when the Reports team subscribes, nothing in Orders changes. Google Pub/Sub is one common example; Apache Kafka and Amazon Kinesis are common streams.

Streams: the topic is a log

In a stream such as Apache Kafka, a topic is a log: an append-only list of events. Reading doesn't remove an event; events are kept for a set retention time, say 7 days. Each reader keeps an offset, the position of the next event it will read, saved where a restart can find it. A short run with two readers:

what happens                  log                    Stock  Email
Orders publishes 2 events     7 new, 4 new           0      0
Email reads both              7 new, 4 new           0      2
Stock reads 1, then crashes   7 new, 4 new           1      2
order 7 paid; Email reads it  7 new, 4 new, 7 paid   1      3
Stock restarts, reads 2       7 new, 4 new, 7 paid   3      3

A reader that crashes resumes from its saved offset, and nothing is lost while it's down. A new reader can start at offset 0, the oldest event kept, and replay history: Reports builds last week's numbers from the log, and a service with a bug can be fixed and replay what it got wrong.

Retention costs some disk. Say orders average about 300 events a second (busy hours are about 5× that, 1,500). With a day of about 100,000 seconds (a quick estimate), that's about 30 million events a day; at about 500 bytes each, 15 GB a day, about 105 GB for 7 days, about 315 GB with 3 copies.

Partitions: how a topic scales

One log read by one server is only as fast as that server. At about 1,500 events a second, a Stock server that handles about 1,000 falls 500 further behind every second. Two servers can't share one log: one offset points at one event, so both would read it.

So a topic is split into partitions, each a log of its own. Each event's key picks its partition: real systems hash the key and take the remainder by the number of partitions (Amazon Kinesis calls partitions shards). With the order id as the key and the id itself as the hash, partition = order id mod 2:

event    order id   id mod 2   partition   offset
7 new    7          1          1           0
4 new    4          0          0           0
7 paid   7          1          1           1
6 new    6          0          0           1
4 paid   4          0          0           2
5 new    5          1          1           2

Each partition numbers its offsets from 0; order 7's events sit in partition 1 in the order published.

Consumer groups

A service's servers form a consumer group, which gives each partition to exactly one of its servers: Stock 1 reads partition 0, Stock 2 partition 1, about 750 events a second each. Every group gets every event and keeps its own offsets: Reports, a group of its own, reads both partitions without moving Stock's offsets. Copies go between groups; work is shared inside one.

So the number of partitions is the most servers a group can use. With 2 partitions, a third Stock server sits idle even at 3,000 events a second, 1,500 for each busy server; with 4, four servers could take 750 each. Choose enough when you create the topic: adding partitions later sends many keys to a different partition. When a server dies, the group hands its partitions to the others, which carry on from the saved offsets.

Per-key order

Inside a partition, events are read in the order they were added. Across partitions there's no order: Stock 1 and Stock 2 go at their own pace, and even Reports, one server reading both, can read order 4's events before new order 7, though 7 came first. Because the key picks the partition, one key's events are read in order: per-key order. So the key is the id whose events must stay in order, here the order id, so "paid" never comes before "new". A key far busier than the rest makes a hot partition, like a hot key in sharding.

Delivery: at least once

A server saves its offset after handling events, so a crash can come in between. Say Stock's saved offset on partition 1 is 2, and Stock 2 sets aside the items for orders 5 and 9, then crashes before saving. The server that takes over starts at offset 2 and handles both again. Like a queue, a topic delivers each event at least once, so consumers must be idempotent: skip what they've already handled, for example orders whose items are set aside. Saving before handling instead loses events in a crash (at most once). Some streams offer exactly-once processing inside the stream, but an email or a card charge still needs an idempotent consumer.

Queue, pub/sub or stream?

• Queue: each job goes to one worker, then it's deleted. For work done once: resize a photo, send one email.
• Pub/sub: each event goes to every subscriber, so several services can react to it.
• Stream (a log): pub/sub that keeps events, with per-key order and replay. The costs: order only inside a partition, a partition count that's hard to change, disk for retention.

Pub/sub and streams in system design interviews

Draw a topic whenever several services react to the same event: an order placed, a post published, a ride ended. A strong answer names the topic, the partition key and why ("orders, by order id, so each order's events stay in order and orders spread evenly"), a consumer group per service, idempotent consumers, and retention for replay. Avoid one queue that every service reads (they split the events), a key by time (a hot partition, an order's events split up), and a producer that calls every service itself.

Common interview questions

What's the difference between a message queue and pub/sub?

A queue gives each message to one reader, so workers share the work. Pub/sub gives every subscriber a copy, so several services react to one event. A stream like Kafka does both: copies between consumer groups, shared work inside one.

How does Kafka keep messages in order?

Only inside a partition. Events with the same key share a partition and are read in the order written, so pick a key whose events must stay in order.

How many partitions should a topic have?

At least as many as the most servers any consumer group will need, since a partition is read by one server of a group. Leave room to grow: adding partitions later moves many keys.

What happens when a consumer crashes?

Its group hands its partitions to the other servers, which start from the saved offsets. Events handled after the last save come again, so consumers must be idempotent.

Summary

• Pub/sub: publish once to a topic; every subscriber gets a copy, and new ones don't change the producer.
• The log: events stay for the retention time; each reader keeps its own offset, to resume or replay.
• Partitions: the key picks one; a group gives each partition to one server, so partitions cap a group's servers.
• Per-key order: one key's events share a partition and stay in order; there's no order across partitions.
• At least once: a crash before the offset is saved repeats events, so consumers are idempotent.