Podcast Hosting

The problem

"Design a podcast hosting service." Creators upload episodes, and the host publishes each show as an RSS feed: a file listing every episode with a link to its audio. Podcast apps, made by other companies, check the feeds of the shows their listeners follow, usually every hour, and download new episodes on their own. Advertisers pay by the download.

This practice problem reuses the pattern of Design YouTube; what changes is the feed, the size of the files and the counting. Park video podcasts, comments and ratings.

The numbers

From the interviewer: 360 million subscriptions (an app following a show), each checked every hour. A feed of 200 episodes is about 200 KB, and each poll gets it whole.

what          working                  result
polls         360 million ÷ 3,600 s    100,000 a second
whole feeds   100,000 × 200 KB         20 GB a second

That is 160 Gbps around the clock, almost all of it unchanged feeds. Two common slips: 72 TB a second puts an hour of polls into one second, and 100 MB a second assumes a poll gets only the newest episode.

The feed: answer 304 Not Modified

Build each feed once, when an episode is published, and cache it at the CDN with an ETag, a tag that changes whenever the feed does. An app sends the tag of the copy it has; if it still matches, the answer is 304 Not Modified: a few hundred bytes of headers and no body.

GET /feeds/s42.xml
If-None-Match: "v17"
  → 304 Not Modified        nothing new, no body

GET /feeds/s42.xml
If-None-Match: "v17"
  → 200 OK   ETag: "v18"    the whole feed, with episode 42

Apps that keep a date instead send If-Modified-Since, and the same rule applies. Say 1 poll in 100 finds a new episode and a 304 is about 500 bytes: 1,000 full feeds a second (200 MB) plus 99,000 small answers (about 50 MB) is about 250 MB a second instead of 20 GB, 80 times less. Without the cache, every poll would also rebuild its feed from the database: 100,000 a second, 20 times what one database server answers (about 5,000).

Asking apps to poll once a day doesn't work: RSS has a hint for it, but the apps aren't yours and many ignore it, and listeners would wait up to a day. Listing only the newest episode breaks apps that missed a few polls.

Audio files: one or two bitrates

An hour of uncompressed WAV, as a creator might upload it, is about 635 MB (1,411 kbps). Speech sounds fine as an MP3 at 128 kbps: 3,600 s × 128 kbps ÷ 8 = about 58 MB, about 40 times smaller than an hour of 5 Mbps video (2.25 GB). On a 5 Mbps phone link it arrives in about 90 seconds, about 40 times faster than it plays.

So YouTube's ladder of 5 renditions in 10-second segments isn't needed: there is no connection speed to adapt to, and a feed names one file per episode. The worker makes one 128 kbps MP3, maybe a 64 kbps copy (about 29 MB) for slow connections, and the original stays in storage.

Publishing: the file before the feed

A hit show has 2 million subscribers whose apps download a new episode within the hour they find it in the feed: about 120 TB in that hour. The feed is the starting gun, so the order matters:

1. The creator uploads the WAV straight to storage with a signed URL.
2. A worker makes the MP3 files.
3. The MP3 is pushed to the CDN's edges: say 1,000 edges × 58 MB, about 58 GB sent ahead.
4. At the release time, the feed gets the episode and a new ETag, and the CDN's cached copy of the feed is purged.
5. Apps polling after that get the new feed and download the MP3 from an edge near them.

A feed that names a file before it's ready sends millions of apps to an error; a feed changed without a purge keeps being answered with 304 and the old copy.

Counting downloads

The host counts downloads from the CDN's logs, and advertisers pay by them. Counting every request overcounts: apps fetch a file in parts and retry, and bots fetch everything. Counting each address once undercounts: an office or a phone network puts many listeners behind one address. A common rule is one download per address + user agent (the app) + episode per day, leaving out known bots. On a small log with two parts of one download, a browser at the same office, a bot, a repeat at 13:00 and a download the next day, requests give 6, addresses give 2 and the rule gives 3.

Many hosts also insert ads at download time (dynamic ad insertion): the current ads are stitched into the episode's audio as the file is served, so old episodes carry this month's ads.

Same pattern as YouTube

The upload path carries over: the service signs a URL, the creator's app sends the file straight to object storage, and a storage event puts a job on a queue for a worker. Episode records live in a database, the files in storage, and a CDN sends the bytes from edges near the listeners.

Summary

• Scope: YouTube's upload and CDN, plus RSS feeds, release rushes and download counts.
• Numbers: 100,000 polls a second; whole feeds would be 20 GB a second.
• Feed: built once, cached with an ETag; 304 Not Modified brings it to about 250 MB a second.
• Audio files: one 128 kbps MP3, maybe a 64 kbps copy; no ladder, no segments.
• Publishing: the file on the edges before the feed names it, then a purge of the cached feed.
• Counting: one per address + app + episode a day, without bots; ads stitched in at download time.