Medium

The problem

"Design Medium." Writers draft long articles in an editor that saves as they type, then publish. Readers follow writers, clap for articles they like, and get 3 free articles a month before a paywall asks them to subscribe. Most readers never sign in: they arrive from a search engine and read one article.

This practice problem reuses the pattern of Twitter at Scale for following and the feed, and the CDNs and Caching lessons for the article pages. Four things are new: drafts saved every few seconds, pages read mostly by signed-out visitors, claps that one reader can repeat, and a metered paywall. Park paying writers and responses for later.

The numbers

From the interviewer: 100 million article reads a day and 100,000 new articles a day. Round a day to 100,000 seconds (86,400 really); busy hours run about 5× the average.

• Reads: 100 million ÷ 100,000 = 1,000 a second on average (about 1,160 with 86,400), so about 5,000 a second at busy hours.
• Publishes: 100,000 a day is about 1 a second, so reads outnumber publishes 1,000 to 1.
• Autosaves: say 20,000 writers are typing at busy hours, each draft saved every 5 seconds: about 4,000 saves a second, nearly as many as the reads.
• Storage: 100,000 articles a day at about 30 KB of text is 3 GB a day, about 1 TB a year.

Drafts: save only the change

A long article is about 30 KB of text, around 5,000 words. Sending the whole article every 5 seconds would move 120 MB a second to change a few words, and overwriting the draft leaves no history. Keeping each whole save as a version costs 720 copies, about 22 MB, for one hour of typing.

Instead each autosave sends only what changed since the last save, say 100 bytes: about 400 KB a second in all, 300 times less. The request names the version it builds on:

PATCH /drafts/d7
{"base": 102, "change": {"at": 812, "insert": "Last year, "}}
  → 200 OK   {"version": 103}

The list of changes is the version history. Keep a full copy every 100 versions; any version is the nearest full copy plus the changes after it, at most 99 to replay. A save whose base isn't the latest version, say from a second tab left open, is refused with 409 Conflict instead of erasing newer text.

Publishing: render once, send from a CDN

At 5,000 reads a second, building each page from the draft on every read repeats the same work about 1,000 times for each publish. And a signed-out reader from a search engine gets exactly the same page as everyone else. So publishing renders the article once into a static HTML page and stores it in object storage, and a CDN sends it from edges near the readers. Search engines get finished HTML instead of a script to run.

When the writer edits and republishes, render again and purge the old page from the edges; the address stays, since search results link to it.

Claps: one request, capped at 50

Say the interviewer allows up to 50 claps per reader on one article. Sending each tap as "+1" makes 12 writes for 12 taps, counts a retried tap twice, and can't stop a 51st clap. Instead the app waits until the reader stops tapping and sends her new total once:

PUT /articles/a42/claps   {"claps": 12}
  → 200 OK   {"yours": 12, "total": 65}

A row (reader, article) holds her claps, capped at 50, and the article's total grows by new − old, so a retry adds 0, like a changed vote in Reddit. The total is a counter per article, as in Distributed Counter: kept ready, and batched in memory when an article goes viral.

The paywall: a meter per reader

The CDN sends one copy of a page to everyone, but the paywall is per reader. Hiding the text with a script fails, since it's already in the page; building pages per reader brings back 5,000 builds a second. So the CDN page holds the opening, and the app asks the servers for the rest with the reader's id: the user id when signed in, a cookie id when not. Members skip the check.

The meter keeps a set of article ids per reader and month, in memory: about 5,000 lookups a second at busy hours, a small share of the 100,000 one in-memory server answers. Ana, signed out as cookie k81, opens a1, a7, a1 again and a9: three free articles, the re-read not counted. Her fourth new article, a3, gets the paywall; on November 1 the key changes and a3 is free.

reader · month   articles
k81 · Oct        a1, a7, a9
k81 · Nov        a3

A cookie meter is soft: clearing cookies starts a new one. Signing in ties the meter to the account, on every device.

Same pattern as Twitter at Scale

Publishing puts a "new article" message on a queue, and workers push the article's id into each follower's ready feed list (fan-out on write); a writer with millions of followers is pulled at read time and merged in. Articles are far rarer than tweets, about 1 a second, so at say 200 followers each the pushes come to about 200 a second.

Summary

• Scope: long drafts, publishing, claps, a metered paywall; follows and a feed as in Twitter.
• Data: drafts as changes plus a full copy every 100 versions; a clap row per reader.
• API: PATCH /drafts/{id}, POST /drafts/{id}/publish, PUT /articles/{id}/claps.
• Numbers: about 5,000 reads a second at busy hours, 1,000 for each publish; 4,000 autosaves a second.
• Services: render once at publish into object storage; a clap counter; a paywall meter.
• Scale: the CDN carries the reads; feeds fan out on write.
• Paywall: the opening from the CDN, the rest through the meter: 3 new articles a month per user id or cookie.