Discord

The problem

"Design Discord." Communities of up to hundreds of thousands of members chat in text channels, scroll back through years of history, drop into voice channels to talk, and use roles to decide who can see and post where. Discord calls a community a "server"; here a server always means a machine.

This practice problem reuses the pattern of Design Chat (WhatsApp). Four things are new: channels far bigger than any Chat group, years of history in one busy channel, live voice, and roles checked on every message. Park streaming a game to friends and paid subscriptions: each is a design of its own.

The numbers

From the interviewer: a big community has 500,000 members, all in #general, and at busy hours #general gets about 20 messages a second. About 50,000 members are online then, and about 5,000 have #general open on screen.

who gets each message      deliveries a second
every member               500,000 × 20 = 10 million
every online member        50,000 × 20 = 1 million
members viewing #general   5,000 × 20 = 100,000

For storage, say #general keeps about 400,000 messages a day (4 a second on average, a fifth of the busy rate), 500 bytes each: 200 MB a day, about 70 GB a year.

Fan-out: only to the members who are looking

Sending every message to every member is 10 million deliveries a second for one channel, and most of those members aren't even online. Sending to everyone online is still 1 million, mostly to people reading other channels. So a message goes in full only to the members viewing the channel: 100,000 deliveries a second, 100× less.

The mechanism is the one from Live Comments: a pub/sub topic per channel. The chat service saves the message and publishes it once; each gateway subscribes to #general's topic while at least one of its members has #general open, and writes the message into those members' WebSockets. Other online members only need to know that #general has something new, so they get one small unread mark, not every message. Offline members get nothing; their app loads the latest messages when they open the channel. A member who is @mentioned is the exception: they also get a push notification.

Storage: partitioned by channel and time

Messages live in a store spread over many nodes by a partition key, and nearly every read is "the 50 messages before this one, in this channel".

• The channel alone: a channel's history sits together, but #general's partition grows by about 70 GB a year, forever, on the same few nodes.
• The community: worse, one huge community becomes one huge partition carrying all its channels' traffic.
• The channel and a time bucket: one partition per channel per 10 days, say. One of #general's buckets holds about 2 GB, a quiet channel's just a few messages, and scrolling up reads the newest bucket, then the one before.

Inside a bucket, messages sort by their time-ordered ids, as in Chat. Discord has described partitioning messages by channel and time. The cost is small: in a quiet channel, a read may step back through a few empty buckets before it finds 50 messages.

Voice channels: UDP through voice servers

Voice is a nonstop stream of tiny slices of sound, and a slice that arrives late is useless. As in Networking Basics, that calls for UDP: a lost packet is skipped, not waited for. On the WebSocket, which runs on TCP, one lost packet would hold up all the sound behind it.

Each member sends one stream to a voice server, which forwards it to the others in the channel: with 10 people, each uploads 1 stream and receives 9. Straight from app to app, each would upload 9 copies (24 with 25 people) and learn everyone's IP address. Voice servers are their own service, placed near the members, because sound flows nonstop while text comes in bursts. The WebSocket still carries the voice channel's news: who joined, who left.

Roles: checked on every message

Roles decide who can see and post where: @everyone may see and post in #general, Mods may see #mods, Admins may post in #announcements. The server checks every post (may Ana post here?) and every delivery (may this member see this channel?), so when a mod takes away Ben's role, the next #mods message skips him. Checking once at join misses role changes, and hiding channels in the app still sends their messages to everyone.

Reading roles from a database for 5,000 viewers × 20 messages would be 100,000 reads a second, so the chat service keeps each community's roles and members in memory and works out each member's permissions from them.

Same pattern as Chat (WhatsApp)

The delivery path carries over. Each open app keeps a WebSocket to a gateway server, the chat service saves a message first and then sends it on, and message ids sort by time. What changes is who it goes to: a Chat group looks up each of its members, up to 100, while a Discord channel publishes once to a topic that only the gateways with viewers subscribe to.

Summary

• Scope: huge communities, years of history, voice channels, roles.
• Data: messages partitioned by channel and time bucket.
• API: POST /channels/{id}/messages; a WebSocket for events; UDP for voice.
• Numbers: 10 million deliveries a second if sent to all; 100,000 to the viewers.
• Services: voice servers apart, forwarding sound over UDP.
• Scale: a pub/sub topic per channel, subscribed while someone views it.
• Roles: checked on every post and delivery, kept in memory.