Message Queue (Kafka)

What the interviewer is asking

"Design a distributed message queue" (often "design Kafka") isn't about using a queue. It's about building one: how a partition is stored on disk, how many servers the load needs, and what keeps a confirmed message alive when a server dies.

Most interviewers have a log in mind, as in Apache Kafka: a topic is split into partitions, each an append-only log where every message gets an offset. Consumers keep their own offsets, so messages stay after they're read. The walkthrough follows the 8 steps of the interview framework.

1. Scope: what version 1 must do

• Append: producers add messages to a topic.
• Read in order: consumers read each partition in order, each at its own pace.
• Replay: any message from the last 7 days can be read again.
• Durable: a server can die without losing a message it confirmed.

The interviewer adds many topics and a heavy, steady load. Name the extras and park them: exactly-once transactions across partitions, a schema registry, and tiered storage that keeps old messages on cheap storage for months.

2. Data: a log of segment files per partition

The servers that store the log are called brokers. A busy partition takes in terabytes a week, and its oldest messages must go every day. One big file won't do (cutting off its front means rewriting everything behind it), and memory is far too small.

So each partition is a chain of segment files of about 1 GB, and only the newest, the open segment, is written. With 1 KB messages a segment holds 1,000,000 of them. Each file is named by its first offset (Kafka pads it to 20 digits, like 00000000000002000000.log):

segment file   first offset
segment 1      0
segment 2      1,000,000
segment 3      2,000,000
segment 4      3,000,000 (open)

Beside each segment sits an index from offset to byte position, sparse (an entry every few KB). Deleting old messages is just deleting old files. The brokers also store each consumer group's committed offsets.

3. The API: produce, fetch, commit

produce(topic, key, value)                  → (partition, offset), once confirmed
fetch(topic, partition, offset, max_bytes)  → messages from offset on, up to max_bytes
commit(group, topic, partition, offset)

The key picks the partition (a hash of the key, modulo the number of partitions), so one key's messages stay in order. The consumer passes its offset: replay is just a fetch from an older one, and the broker keeps no position for any reader beyond the offsets it commits. Max bytes lets one fetch bring back thousands of messages instead of one round trip each. A consumer commits after handling messages, so a crash repeats a few rather than skipping any: at-least-once.

4. The busy path: produce and fetch

1. The producer hashes the key ana to pick its partition, P0.
2. It sends the message to the broker that leads P0 (section 6).
3. The broker appends it at the end of P0's open segment, at the next offset.
4. It answers with (P0, offset) once the message is confirmed.
5. A consumer reading P0 gets it with its next fetch.

Fetching mirrors it. A consumer at offset 2,350,000 asks for P0; a binary search over the sorted names finds the last file that starts at or before it: segment 3, which starts at 2,000,000 (segment 4 starts at 3,000,000). The index points to the message, and the broker reads on, up to max bytes, in one sequential read.

5. The numbers

From the interviewer: 1 million messages a second, about 1 KB each, kept 7 days in 3 copies. For a quick estimate a day is about 100,000 seconds (86,400 really).

• In: 1 million × 1 KB = 1 GB a second.
• Storage: 1 GB × 100,000 seconds ≈ 100 TB a day; 7 days ≈ 700 TB; 3 copies ≈ 2,100 TB, about 2 PB (1.8 PB with 86,400 seconds). Forgetting the copies gives 700 TB; keeping one day gives 300 TB.
• Disk writes: every copy is written somewhere: 3 GB a second. Say a broker writes about 500 MB a second when it only appends: 3 GB ÷ 500 MB = at least 6 brokers. Counting each copy as a database write, about 5,000 a second, gives 3 million ÷ 5,000 = 600: the wrong model (section 8).
• Network: each of the 6 brokers takes in about 500 MB a second, half of a server's network of about 1 GB a second.
• Disk space: 2,100 TB on 6 brokers is 350 TB each, so disk space, not speed, often sets the broker count.

6. Services: brokers, leaders and a controller

Each partition lives on 3 brokers, as in the Replication lesson: one copy is the partition's leader, which takes its writes and reads, and the other two follow it. Spread the leaders out:

broker     P0        P1        P2
Broker 1   leader    follower  follower
Broker 2   follower  leader    follower
Broker 3   follower  follower  leader

Each broker does a third of the work, and any one can die while the others hold every partition. Clients ask any broker which broker leads each partition, then talk to the leaders directly: no router sits in the path of 1 GB a second.

The followers can't pick a new leader on their own: two could both take over P0, the split brain from the Consensus and Leader Election lesson. A controller, elected by a majority vote, assigns every leader, watches the brokers' heartbeats and tells every broker. Newer Kafka elects it among a few controllers with Raft (KRaft); older versions used ZooKeeper. Each new leader gets a higher leader epoch, like a Raft term.

7. Scale: how many partitions

A partition goes to one server of a consumer group. Say the busiest topic, clicks, brings in 500 MB a second, and one consumer server processes about 50 MB a second: 500 ÷ 50 = 10 partitions at least, and on 6 brokers each leads 1 or 2. Pick a few more, because adding partitions later sends keys to other partitions.

Each partition then takes 50 MB a second: a 1 GB segment fills every 20 seconds, and 7 days come to about 35 TB. That's what moving one partition to a new broker has to copy.

8. Deep dive: why appending is fast

Each broker writes about 500,000 messages a second. Recall the latency numbers: memory about 100 ns, an SSD read about 100 µs, a disk seek about 10 ms. A spinning disk that seeks for every write manages about 100 writes a second, which is what an index updated at random places costs.

A log never seeks to write: every message goes at the end of the open segment, so the disk streams at full speed, and SSDs too do best with big sequential writes. Producers send batches, so one write carries many messages, and consumers read big chunks. Kafka leaves caching to the operating system's page cache, so readers that keep up are served from memory. Brokers don't force each write onto the disk; durability comes from the copies on other brokers.

9. Deep dive: replication and acks

Followers fetch new messages from the leader. The leader and the followers that keep up are the in-sync copies; one that falls behind for too long (say 30 seconds) is dropped, so it can't hold up every write. With acks=1 the leader confirms as soon as it has appended; with acks=all, only once every in-sync copy has the message, at least one round trip inside the data center (about 0.5 ms) later. Run P0 with acks=1:

step                       Broker 1    Broker 2   Broker 3   acked
start                      a b c       a b c      a b c      a b c
d sent, acked              a b c d     a b c      a b c      a b c d
Broker 2 gets d; e acked   a b c d e   a b c d    a b c      a b c d e
Broker 1 dies              down        a b c d    a b c      a b c d e

Broker 2 becomes the leader, so P0 holds a, b, c and d; a returning Broker 1 drops e. The producer got an ack for e, and e is gone.

With acks=all, Broker 1 acks d only after both followers have it, and e, on Broker 2 only, isn't acked when Broker 1 dies. Whichever follower leads next, a to d are there. The producer sends e again; Kafka's idempotent producer numbers its messages, so a leader that has e drops the repeat. Two more rules complete it: writes are refused while fewer than 2 copies are in sync, and consumers only see messages every in-sync copy has (the high watermark).

10. Deep dive: leader failover

1. Broker 1 stops sending heartbeats to the controller.
2. After a timeout, the controller marks it dead.
3. For P0, which Broker 1 led, it picks a new leader from P0's in-sync followers: Broker 2.
4. It tells all the brokers.
5. Producers and consumers fetch fresh metadata and learn the new leader.
6. They carry on with Broker 2, at the offsets they had.

Picking a follower that fell behind would bring P0 back sooner but lose messages; Kafka calls that an unclean leader election and keeps it off by default. P1 and P2 only lose a follower copy, and Kafka doesn't copy it elsewhere by itself: they stay one copy short until Broker 1 returns or an operator moves them.

11. Deep dive: retention and compaction

Time-based retention deletes whole segments once their newest message is over 7 days old. But a topic of each user's settings must give a new service everyone's latest value, even one set a year ago. For such topics, log compaction keeps the latest message per key:

offset   before        after compaction
0        ana: dark     –
1        bo: light     –
2        ana: light    ana: light
3        cy: dark      cy: dark
4        bo: dark      bo: dark

A background job rewrites old segments; offsets stay as they were, with gaps, and a message with an empty value (a tombstone) deletes its key. Kafka keeps the groups' committed offsets in a compacted topic of its own.

What breaks first

• A hot key: all its messages go to one partition and overload one leader. Spread it over partitions only if its order doesn't matter.
• Disk space: 350 TB per broker. Tiered storage (version 2) moves old segments to object storage.
• A whole rack failing: put each partition's 3 copies in 3 racks; Kafka can place copies by rack.
• Duplicates: retries repeat messages, so use the idempotent producer and idempotent consumers.

Summary: the order to follow

1. Scope: append, read in order, replay 7 days, never lose a confirmed message.
2. Data: 1 GB segment files per partition, named by their first offset, each with an index.
3. API: produce, fetch from an offset up to max bytes, commit.
4. Busy path: key → partition, one append at the end of the open segment.
5. Numbers: 1 GB a second in, about 2 PB stored, at least 6 brokers.
6. Services: leaders spread over the brokers, assigned by an elected controller.
7. Scale: enough partitions for a consumer group: 10 for clicks.
8. Deep dives: batched appends; acks=all; failover to an in-sync follower; retention or compaction.

The same steps work for any "design X" question.