ChatGPT

What the interviewer is asking

"Design ChatGPT" means design a chat assistant: people send messages to a large language model (LLM) and read its answers as they're written. The hard part isn't the chat: every answer is written by a model on GPU servers, which are costly and scarce, so the design is about getting the most out of each GPU while keeping answers quick and fair. The walkthrough follows the 8 steps of the interview framework: scope, data, API, the busy path, numbers, services, scale and deep dives. It describes a generic service, not any company's real system.

1. Scope: what version 1 must do

• Send a message and get the model's answer.
• See the answer stream word by word, as it's written.
• Keep conversations and continue one later.
• Fair-use limits for each user.

Park image input, tools such as web search, voice, and training: training is a separate, offline system, and this design serves a trained model. Mention safety in one line: a moderation step checks messages and answers, a topic of its own.

2. Data: conversations, messages, usage

record         fields
conversation   id, user, title, created
message        id, conversation, role (user or assistant), text, tokens
usage          user, tokens this minute, tokens today

A token is a short piece of text, a word or part of one; in English a token averages about 4 characters. Saving each message's token count lets the history be trimmed to fit the model without counting again. Conversations and messages go in a database, split by conversation id as it grows; usage counts are checked on every request, so they live in a fast in-memory store.

3. The API: a stream of tokens

GET  /conversations
  → 200 OK   [{"id": "c7", "title": "Recursion help"}, …]

GET  /conversations/c7
  → 200 OK   {"messages": […]}

POST /conversations/c7/messages   {"text": "Show me Python code"}
  → 200 OK   Content-Type: text/event-stream
  data: {"token": "Here"}
  data: {"token": " is"}
  …
  data: {"done": true}

A long answer takes the model about 10 seconds to write, so the POST doesn't wait for it. It answers with a stream of server-sent events: one ordinary HTTP request whose response stays open and carries each token as it's written, the use the Real-Time Updates lesson names for an AI assistant's answer. One JSON reply at the end would leave the user staring at a spinner; polling would bring the words in jumps. A WebSocket works too, but this stream flows one way.

4. The busy path: one message, token by token

1. The app sends POST /conversations/c7/messages.
2. The chat service saves the message and loads the conversation so far.
3. A model server reads the whole conversation.
4. It writes the answer one token at a time, say 50 tokens a second, and each token is streamed to the user at once.
5. When the answer ends, the chat service saves it.

Saving and loading take milliseconds; writing takes seconds, since each token means running the whole model once, billions of multiplications. That's why model servers run on GPUs, chips that do thousands of multiplications at once and keep the model in their own fast memory. CPU servers would write each answer many times slower, and a large model needs tens to hundreds of GB of memory, often several GPUs' worth, so one model server is a machine with several. GPUs are the scarce, costly resource.

5. The numbers

From the interviewer: 10 million daily users, about 10 messages each a day, answers of about 500 tokens. Busy hours run about 5× the average; round a day to 100,000 seconds.

• Messages: 10 million × 10 = 100 million a day ÷ 100,000 = 1,000 a second (about 1,200 with the exact 86,400), so 5,000 a second at busy hours.
• Tokens: 5,000 answers a second × 500 tokens = 2.5 million tokens a second at busy hours.
• Model servers: say one server writes about 1,000 tokens a second in all, batching many conversations. 2.5 million ÷ 1,000 = 2,500 model servers, plus spares.
• Everything else: 5,000 messages a second at about 1,000 requests a second per app server is about 5 chat servers. Storage: about 2.5 KB for a message and its answer, so about 250 GB a day, 90 TB a year before copies.

The ratio is the point: about 2,500 GPU servers against a handful of ordinary ones.

6. Services: gateway, chat service, model servers

• API gateway: checks sign-in and each user's limits before anything costly runs.
• Chat service: owns the history: saves messages, loads conversations, builds what the model reads.
• Model servers: only run the model. They keep no history and never touch the database.

Each grows on its own: a few gateway and chat servers behind load balancers, about 2,500 model servers. Tokens flow back the way the request came, to the user's open stream.

7. Scale: an inference queue

At busy hours about 5,000 answers start every second across 2,500 model servers. Each server works on a few dozen conversations at once, and answers last from a second to a minute, so some servers are full while others have room. Round robin can't see that, and random picks with retries turn a busy minute into a storm of retries. Instead, the chat service puts each request on an inference queue, and each model server pulls the next request when its batch has room: a queue in front of scarce workers, as in Message Queues. A burst waits in the queue instead of piling onto full servers.

8. Deep dive: streaming

Say the model server takes about 0.5 s to read the conversation, then writes 50 tokens a second, and each token's trip to the screen takes about 0.1 s. For a 500-token answer:

event                         sent at the end   streamed
the model reads the chat      0 to 0.5 s        0 to 0.5 s
first token made              0.52 s            0.52 s
first word on screen          10.6 s            about 0.6 s
last token made               10.5 s            10.5 s
whole answer on screen        10.6 s            10.6 s

Streaming doesn't make the answer finish sooner: it makes the wait about 0.6 s instead of 10.6 s. Time to first token is the number users feel, and it grows with the conversation the model must read first.

9. Deep dive: batching on the GPU

A model server writes tokens in steps: each step runs the model once and gives every conversation in its batch one more token. Most of a step goes into reading the model from GPU memory, which costs about the same for one conversation or many, so a bigger batch makes far more tokens for a little more time per step. Say:

batch   one step   tokens a second   each conversation   a 500-token answer
1       20 ms      50                50 a second         10 s
8       25 ms      320               40 a second         12.5 s
32      32 ms      1,000             about 31 a second   16 s

A batch of 32 makes 20× the tokens of a batch of 1 from the same server: 2,500 servers instead of 50,000 (a batch of 8 would need about 7,800). Each answer streams a little slower, which nobody notices while reading, as long as it stays above what the product promises, say 30 tokens a second. Counting the other way gives the same fleet: 5,000 answers start every second and each takes 16 s, so 80,000 are being written at once, 32 on each of 2,500 servers.

Servers use continuous batching: a new request joins the batch between two steps and a finished answer leaves at once, instead of the whole batch waiting for its longest answer. The batch size is limited by GPU memory, since each conversation in the batch keeps its working state there.

10. Deep dive: conversation history

The model remembers nothing between requests: serving never changes it, and the next message may land on any server. So the chat service sends the whole conversation so far with every request, each earlier turn with its role, then the new message. The history database is the memory; the model only reads.

That makes long chats costly. Say a turn, a message and its answer, is about 600 tokens: after 20 turns, the next message would send about 12,000 tokens. A model reads at most a fixed number of tokens at once, the answer included: its context window, from a few thousand tokens to a million or more in today's models; say 8,000 here. A common fix is a summary of the older turns, then the newest turns that fit:

part sent                      tokens
summary of turns 1–9           400
turns 10–20, word for word     6,600
the new message                100
left for the answer            900 of 8,000

Sending only the new message forgets the chat; keeping the first 8,000 tokens drops the recent turns. The model can write the summary once, stored with the conversation. Keep every message in the database, and keep the recent turns of active conversations in an in-memory cache, since every new message reads them (Caching).

11. Deep dive: queueing and limits

Some evenings demand beats even 2,500 servers, and the inference queue grows. Three tools keep it fair:

• A timeout in the queue: a request that has waited, say, 30 seconds leaves with a "busy, try again" error (503) instead of an endless spinner, and no GPU is spent on someone who has left.
• Tokens a minute per user, checked at the gateway: a limit on requests would let ten huge prompts through, so count the tokens each request uses (Rate Limiting, counting tokens instead of requests). Over the limit, the gateway answers 429 with Retry-After.
• Priority: paid plans' requests go to the front of the queue. Keep a share of the servers for free users, so at peaks they're slower but never starved.

What breaks first

• GPU supply: you can't add GPU servers in an afternoon. Plan ahead of known peaks, and degrade gently, for example with a smaller, faster model for free users.
• A model server dying mid-answer: the chat service starts the answer again on another server, or offers to regenerate it.
• Very long chats: time to first token grows with what the model reads, so summarize early.
• The history database growing: about 90 TB a year. Split it by conversation id and move old chats to cheaper storage.

Summary: the order to follow

1. Scope: chat with a model, answers streamed, history kept, limits per user.
2. Data: conversations, messages with role and tokens, usage counts.
3. API: POST /conversations/{id}/messages answers with server-sent events; two GETs read the history.
4. Busy path: the model reads the whole chat, then writes the answer token by token.
5. Numbers: 2.5 million tokens a second at busy hours, about 2,500 model servers.
6. Services: a gateway, a chat service with the history, model servers on GPUs.
7. Scale: an inference queue the model servers pull from when they have room.
8. Deep dives: streaming (the first word in about 0.6 s), batching (32 conversations a step, 1,000 tokens a second), history (resent each time, summarized past the context window), limits (timeouts, tokens a minute, paid plans first).

The same shape fits any service built around a scarce, slow worker, such as image generation: queue the requests, batch them, stream what you can.