Image Generation Queue

The problem

"Design an image generator like Midjourney." A user types a prompt, such as "a fox in the snow", and gets back a picture made by an AI model on GPU servers. Each image keeps one GPU busy for about 5 seconds, and nothing comes out until the picture is finished: unlike a chat answer, there are no words to stream.

This practice problem reuses the pattern of Design ChatGPT with the job queue of Design LeetCode. What's new: waiting for a whole image, batching that trades seconds for GPUs, priority lanes, big files to keep, and a check on the picture itself. Park video, a public gallery and editing.

The numbers

From the interviewer: about 1,000 images a second at busy hours, each taking about 5 seconds of one GPU.

• GPUs busy at once: 1,000 new images a second, each holding a GPU for 5 s, means 1,000 × 5 = 5,000 GPUs running at any moment.
• Cost: say a GPU rents for about $2 an hour: 5,000 GPUs cost about $10,000 an hour at the peak. GPUs are both the bill and the limit.
• Images: with busy hours at about 5× the average, the average is about 200 a second, about 20 million a day (a day rounded to 100,000 seconds; about 17 million with 86,400). At about 1 MB each, that's 20 TB a day, and 1 GB a second written at the peak.

Waiting: a job id and a place in line

POST /images   {"prompt": "a fox in the snow"}
  → 202 Accepted   {"job": "j42"}

GET /jobs/j42   → {"status": "queued", "place": 12}
GET /jobs/j42   → {"status": "running", "progress": 60}
GET /jobs/j42   → {"status": "done", "image": "https://cdn…/j42.png"}

A free user's job may wait minutes before a GPU takes it, so POST /images answers 202 Accepted with a job id at once, as LeetCode's submissions do. The API service saves the job in a jobs database and puts its id on the queue; the app asks for the status every few seconds, or the server pushes "done" over a WebSocket or server-sent events. A closed tab loses nothing.

The place in line works like tickets at a deli counter: each lane numbers its jobs and remembers the last number a GPU took, so ticket 1,012 with 1,000 taken is 12th. Holding the request open would tie up thousands of idle connections, and there's nothing to stream: the model works on the whole picture at once.

Batching: fewer GPUs, slower images

Say one image takes 5 s and a batch of 4 prompts takes about 10 s on one GPU. Then 1,000 images a second make 250 batches a second, each holding a GPU for 10 s: 2,500 GPUs instead of 5,000, half the bill. The price: each image takes 10 s instead of 5, small next to a wait in line. Dividing 5,000 by 4 (1,250) forgets that a batch takes longer than one image. A batch groups prompts of the same image size; past a point, a bigger one saves little.

Priority lanes and caps

At 9 p.m. jobs arrive faster than the GPUs clear them. Give the queue two lanes: GPU servers take paid jobs first, as LeetCode's runners take contest runs first, and free jobs when the paid lane is empty. A paid job goes ahead of 2,000 free jobs that came earlier; a free job at 9 p.m. waits minutes, one at 4 a.m. starts at once.

Two guards keep it fair. Take every 5th batch from the free lane anyway (say), so free users are never starved for hours. And put a cap per user at the API, as ChatGPT's limits do: say 4 jobs at once for a free account and at most 4 images a request. A script sending 100 prompts a minute gets 429 before it costs a GPU second.

Where the images go

As in Image Hosting, the GPU server writes the finished 1 MB file to object storage, the job's row keeps only its key, and a CDN sends it from edges near the viewers, so a shared image never touches your servers. Storing the bytes in the jobs database would fill it with 20 TB a day and slow every status read; sending them straight to the app fails when the tab is closed and keeps nothing for next month.

Two safety checks

• The prompt, before the job is queued: a small, fast text model refuses a banned prompt in milliseconds, before it costs GPU time.
• The image, before it's stored or shown: a harmless prompt can still give a harmful picture.

Same pattern as ChatGPT, with LeetCode's job queue

From ChatGPT: scarce, costly GPU servers pull work from a queue when they have room, batching gets more out of each, and limits per user put paid plans first. From LeetCode: the job is accepted at once with an id, workers run it in the background, the page asks for the result, and urgent work goes first. What changes: one whole file instead of a stream of tokens, so the wait shows as a place in line and the file lives behind a CDN.

Summary

• Scope: a prompt in, a whole image out, kept for later.
• API: POST /images answers 202 with a job id; GET /jobs/{id} gives the place in line, the progress, then the image.
• Numbers: 1,000 images a second × 5 s = 5,000 GPUs busy at the peak.
• Batching: batches of 4 in about 10 s: 2,500 GPUs, 10 s an image.
• Lanes: paid first, free when there's room and every 5th batch, a cap per user.
• Images: object storage behind a CDN, only the key in the job's row.
• Safety: check the prompt before the queue and the image before anyone sees it.