Pastebin

The problem

"Design Pastebin" often follows the URL shortener. People paste text, up to 10 MB, and get a short link such as pst.bin/k3J9q that anyone can open to read it. A paste can expire after a time its author picks (10 minutes, a day, a month or never), and some are private: only people who have the link can read them.

Most of it is the URL shortener again, so say so and spend the interview on what is new. The API is two calls:

POST /pastes   {"text": "…", "expires_in": "1d", "private": false}
  → 201 Created   {"code": "k3J9q"}

GET /k3J9q
  → 200 OK   the text   (404 Not Found once it has expired)

Unlike a short link, the read answers with the text itself, not a redirect, so the size of a paste matters.

The numbers

A typical set: 1 million new pastes a day, about 10 KB each on average (the biggest 10 MB), each read about 10 times.

quantity           estimate
new text a day     1 million × 10 KB = 10 GB
text a year        about 3.6 TB (about 11 TB in 3 copies)
rows a day         1 million × 500 bytes = 0.5 GB
reads a second     10 million ÷ 86,400 ≈ 120

Two common mistakes: counting the reads as if they stored text (100 GB a day), and giving every paste the biggest size (10 TB a day). The load is small, about 120 reads a second, but the bytes grow fast: the text is about 20 times the size of the rows.

Decision 1: where the text lives

Keep each paste's text in object storage under a key, and a small row in the database: the code, the storage key, the expiry and whether it's private, about 500 bytes. Rows stay small and lookups fast; object storage keeps big objects cheaply, in several copies.

A text column works for tiny pastes, but rows of up to 10 MB make every replica and backup copy all of it, and fill the database's memory with text. A server's own disk is worse: the next read may reach another server, and a dead disk loses its pastes.

Decision 2: viral pastes

A few pastes get most of the reads. Say a 1 MB paste is linked from a popular forum and gets 5,000 reads a second: 5 GB a second of the same bytes, while one server's network sends about 1 GB a second.

The shortener's cache in memory isn't enough. A short link's hit was a few bytes; a paste's is 1 MB, so every byte would still leave your servers, and one paste is one key on one cache server. Read replicas don't help: the database only serves the small row. The fix is a CDN: a paste never changes once made, so edges near the readers keep copies and answer from them.

The catch: an edge's copy doesn't know its paste expired. The paste service sets each copy's lifetime to the time the paste has left, in the Cache-Control header (an hour left: max-age=3600), so when the paste expires the edge asks again and gets 404.

Decision 3: expiry

You need two parts. Every read compares the row's expiry with now and answers 404 once it has passed (410 Gone also fits): expiry exact to the second. A cleanup job runs every hour and deletes what expired, which frees the space; it can run late, because reads already refuse expired pastes.

A job alone lets a paste live up to an hour too long. A read check alone keeps every dead paste forever, in 3 copies. Built-in TTLs and storage lifecycle rules help, but some delete late, so keep the read check.

The cleanup job's order

The job finds a batch of expired rows (an index on the expiry makes it quick), deletes each one's text from object storage, then deletes the rows, and repeats. If it crashes in between, the rows are still there and still expired, so the next run finishes the job; deleting text that's already gone does no harm. Rows first would leave text no row points to, stored and paid for forever.

Decision 4: private pastes

Public codes come from blocks of numbers, so they are easy to walk: paste 1,000,000 gets 4c92, the next one 4c93. Leaving private pastes out of a public list doesn't hide them, and a longer counter code is still in order.

A private paste gets a random code: 16 characters from a secure random generator, as in Unique IDs. That's about 5 × 10²⁸ codes (95 random bits): with a billion private pastes stored, someone trying a million codes a second for a year has less than one chance in a million of hitting one. The code is the row's unique key, so the rare repeat is rejected and the service picks again.

Same pattern as the URL shortener

• A lookup by key: the link's code finds one record, on the busy read path.
• Codes from a counter: public codes are numbers from blocks a shared counter hands out, in base 62, so codes never clash.
• Read-heavy and uneven: far more reads than writes, and a few hot items. The cache becomes a CDN because each read carries the text.

Summary

1. Scope: the URL shortener's pattern, plus 10 MB texts, expiry and private pastes.
2. Data: a small row (code, storage key, expiry) in the database; the text in object storage.
3. API: POST /pastes; GET /k3J9q answers the text, or 404 once expired.
4. Numbers: about 10 GB of new text a day, up to 3.6 TB a year, about 120 reads a second.
5. Scale: a CDN carries viral pastes; each copy lives only as long as its paste.
6. Expiry: every read checks it; an hourly job deletes the text, then the row.
7. Private pastes: 16 random characters instead of counter codes.