Files and Object Storage

The problem: photos inside database rows

The quickest way to add photos to an app is to save each one inside its post's row, as a blob (a binary large object). Most databases have a column type for it, and on day one it works. But a post's row is about 500 bytes, and with a 1 MB photo inside it's about 2,000 times bigger.

Say 1 million photos are posted a day. That's about 1 TB of new data a day, about 400 TB a year, and a database never keeps just one copy:

• Replicas copy every byte, so each one needs 400 TB more disk a year.
• Backups copy it again, and a restore or a new replica must copy all of it before it's ready: at about 1 GB a second, 400 TB takes more than 4 days. The rows alone, about 500 MB a day and under 200 GB a year, copy in about 3 minutes.
• Memory: serving photos through the database fills its memory with photo bytes instead of rows.

How object storage works

Object storage is a store built for files. Each file is an object, kept in a bucket and found by its key, a name like photos/42/9001.jpg. Amazon S3 is a common example; Google Cloud Storage and Azure Blob Storage work much the same way. It keeps each object in several copies, on different machines, and grows to petabytes without you adding servers.

It does a few things: save an object under a key, read it, delete it, list keys by prefix. It can't search inside files, and an object is replaced whole rather than edited in place.

So each post lives in two places. The database row keeps what the app searches, sorts and shows, plus the file's metadata; object storage keeps the bytes:

field          kept in          example
id             database row     9001
user           database row     42
caption        database row     Sunset at the pier
photo key      database row     photos/42/9001.jpg
size, type     database row     1 MB, image/jpeg
thumbnail key  database row     thumbs/42/9001.jpg
the photo      object storage   1 MB of bytes
thumbnail      object storage   a small file of its own

Uploads straight to storage, with signed URLs

The next trap is uploading through your app servers. Say users post 100,000 videos a day, about 100 MB each. A day is about 100,000 seconds for quick estimates (86,400 really), so that's about 1 upload a second, and about 5 a second at the busy hour, 5× the average. Say a phone sends about 1 MB a second: each upload takes 100 seconds, so about 500 are in progress at any moment. Each holds a connection on your app servers the whole time, where a normal request is over in milliseconds, and 500 MB a second flow in from phones and straight back out to storage.

The fix is a signed URL (S3 calls it a presigned URL): a link to storage that carries its own permission. The app signs the method, the key and an expiry time with its secret key; storage recomputes the signature and refuses the request if anything was changed or the time is up. The secret never leaves your servers:

POST /uploads    {"type": "video/mp4", "size": 100000000}
  → 200 OK       {"key": "videos/42/77.mp4",
                  "url": "https://storage.example.com/videos/42/77.mp4?expires=…&signature=…"}

PUT https://storage.example.com/videos/42/77.mp4?expires=…&signature=…
  (100 MB of video)
  → 200 OK

1. The phone asks the app for an upload URL.
2. The app checks the user, picks the key and saves a row marked uploading.
3. The app signs a URL for that key, valid for 15 minutes, and sends it back.
4. The phone sends the bytes straight to object storage, which checks the signature and saves the object.
5. The phone tells the app it's done, and the app marks the row ready. Or storage announces the new object itself: S3, for one, can put a message on a queue for each new object.

Avoid two shortcuts: a bucket anyone can write to, and the storage secret built into the phone app, where anyone who unpacks it finds it.

Big files: chunks and resumable uploads

If the connection drops 65 MB into a single upload, all of it is lost, since storage saves an object only once all of it has arrived, and the retry starts from the first byte. On a train that loses the signal every minute, a 100-second upload may never finish.

So big files go up in chunks. S3 calls it a multipart upload, with parts of at least 5 MB except the last; with 5 MB chunks, a 100 MB video is 20 of them, each sent to its own signed URL. Storage keeps every chunk it receives, and after a drop the phone asks which ones it has and sends only the rest:

moment                       storage has    still to send
start                        nothing        chunks 1–20, 100 MB
chunks 1–13 confirmed        chunks 1–13    chunks 14–20
drop halfway through 14      chunks 1–13    chunks 14–20, 35 MB
reconnect, send 14–20        chunks 1–20    nothing: join them

The phone sends 7 chunks, 35 MB, instead of all 100 MB, so a drop costs at most one chunk, 5 MB: about 102.5 MB sent in all, against 165 MB when the whole file starts over. Each chunk carries a checksum, so storage rejects a damaged one and only that one is sent again, and on a fast link several chunks can go at once. Once all 20 are in, storage joins them into one object.

After the upload: processing off a queue

A new photo needs a thumbnail, a new video smaller copies for slow connections (transcoding). That takes from a second to a few minutes, so it stays out of the upload: storage's new-object event goes on a queue, and workers take jobs off it and write each version back to storage as its own object, with its key in the row. If a worker crashes, the job waits on the queue for another (Message Queues). Design YouTube takes transcoding further.

Private files: signed download links

Public files can go through a CDN as they are. For a private photo, the app first checks that the viewer may see it, then hands out a signed download URL that expires in a few minutes. Many CDNs can check such signatures themselves, so the photo still comes from an edge near the viewer, and a link copied out of the app soon stops working. A public object under a long random key would leak for good once shared; sending every view through your app servers would put them back in the path of the bytes.

Where it shows up in design interviews

Every design with photos, video or documents needs it: Design Instagram, Design YouTube, Design Dropbox. Say where the bytes go and what the database keeps, how uploads reach storage, what happens on a dropped connection and where processing runs, and back it with a number: 500 uploads at once would tie up the app servers, while with signed URLs they answer two small requests per upload.

Common interview questions on file storage

Why not store images in the database?

It can hold them, but every replica and backup then carries every image, restores take days instead of minutes, and photo bytes crowd its memory. Keep the bytes in object storage and the key and metadata in the row.

What is a presigned URL, and why use one?

A link to one object, signed by your server with its secret key, that allows one action, an upload or a download, until it expires. Clients move the bytes straight to and from storage, and your servers only hand out permissions.

How do you upload a very large file reliably?

Split it into chunks, each with a checksum, upload them separately or several at once, and let storage join them at the end. After a dropped connection, ask which chunks arrived and send only the missing ones.

Summary

• Files go in object storage under a key; the database row keeps the metadata and the key.
• Phones upload straight to storage with signed URLs, so app servers never relay the bytes.
• Big files go in chunks: after a drop, only the missing chunks are sent again.
• Processing runs off a queue, and private files go out through signed links that expire.