Image Hosting (Imgur)

The problem

"Design an image host like Imgur." Anyone uploads an image, with no account, and gets a short link such as img.ur/x7Kp2 to share. Most views don't come from your own site at all: forums, chats and news sites put the image's file address, i.img.ur/x7Kp2.jpg, straight into their pages. That is an embed, or hotlink.

This practice problem reuses the pattern of Design Instagram. What changes is everything that depended on accounts: there is no feed and no follows, nobody signs in, a few viral images bring most of the traffic, and deleting has to work for someone the system never identified. Park albums, comments and votes: they come with accounts.

The numbers

From the interviewer: 10 million uploads a day of about 1 MB each, and 2 billion views a day, mostly embeds. Round a day to 100,000 seconds (86,400 really).

• Uploads: 10 million ÷ 100,000 = 100 a second, about 100 MB a second coming in.
• Storage: 10 million × 1 MB = 10 TB a day, about 3.65 PB a year before storage's own copies.
• Views: 2 billion ÷ 100,000 = 20,000 a second, so about 20 GB a second going out (about 23 with 86,400), 100 GB a second at busy hours.

That is 200 views for every upload. The design is about the bytes that go out.

Viral images: where the cost lands

Say a news site puts one meme on its front page: 10,000 views a second for an hour, 36 million views in all. For your servers it is the easy case. The embed address is the file's own key, so a view never touches the image service or its database: the CDN fetches the file from object storage by that key. With 100 edges that each keep a copy for a day, storage sees 100 requests in that hour, one miss per edge.

What doesn't shrink is the bytes: 10 GB a second, half the whole service's average, about 36 TB in the hour, and CDNs bill by the GB. The real misses are the long tail, the millions of images seen a few times a day. So what a viral image costs is bandwidth, not load.

Abuse: a limit for each kind

With no accounts there is no one to ban, so each kind of abuse gets its own limit:

• A rate limit per address. A script uploading thousands of images an hour, or an address trying thousands of random codes a minute to find other people's images, is counted per IP address and answered with 429. An address is a weak key: a whole office can share one, and an attacker can switch.
• A size limit per file. The signed upload can carry a maximum size, say 20 MB, so storage itself refuses a 2 GB video renamed photo.jpg, and workers check that the bytes really are an image. A 150 MB animated GIF as a busy forum's banner would send 150 MB per page view: cap animated files too, and turn big GIFs into a short video, often many times smaller.
• Reports and review. A limit can't judge what an image shows. A report button sends harassment or banned adult images to review, a classifier can flag likely adult images at upload, and a reviewer takes an image down the way a delete does.

The codes work like the URL shortener's, a code from a block of numbers, but shuffled so nobody can count through other people's images. Seven characters give about 3.5 trillion codes; after ten years of 10 million uploads a day, about 1 code in 100 is taken. A link-only image is unlisted, not private.

Deleting without an account

POST /images
  → 201 Created  {"link": "img.ur/x7Kp2",
                  "image": "i.img.ur/x7Kp2.jpg",
                  "delete_token": "k9Qz…"}

DELETE /images/x7Kp2   token: k9Qz…
  → 204 No Content

The upload's answer carries a secret delete token that only the uploader's browser ever sees, and a delete must bring it. The database keeps only a hash of the token, as with passwords. "Anyone with the short link" would include everyone who has seen the image, and "the address that uploaded it" locks out a phone that changed networks while letting in a stranger on the same café Wi-Fi. A lost token means the image stays unless it's reported.

Gone from the CDN too

Deleting the row and the file in storage is not enough: each edge answers from its own copy without asking storage, so the news site would keep showing the image for up to a day. The delete must also purge the file from every edge, which takes seconds to minutes; the next view misses and storage answers 404. Copies in viewers' browsers can't be purged, so give browsers a shorter cache time, say an hour, than the edges' day.

A one-minute copy everywhere would wreck the hit rate instead: the meme would cost 6,000 fetches an hour instead of 100, and an image seen less than once a minute at an edge would miss every time. Deletes are rare; purge those.

Same pattern as Instagram

Everything about moving bytes carries over. The image service signs an upload URL, so the browser sends the file straight to object storage. Storage announces each new file on a queue, and image workers check it, make a thumbnail and strip hidden location data from photos. The CDN sends the bytes from edges near the viewers.

Summary

• Scope: anonymous uploads, a short link, embeds, deletes; no feed.
• Data: code → image file, and a hash of its delete token.
• API: POST /images answers with the link, the image address and a delete token.
• Numbers: about 20 GB a second out, 200 views per upload.
• Scale: a viral image costs one miss per edge; bandwidth is the bill.
• Abuse: a rate limit per address, a size limit per file, reports for the rest.
• Deleting: a token only the uploader has, then a CDN purge.