AI Code Assistant

What the interviewer is asking

"Design an AI code assistant" (think of tools like GitHub Copilot) asks for the gray text that appears in your editor while you type: the next few lines, written by a model, accepted with Tab. It mixes a classic busy read path with what is new in AI systems: a model that is slow and costly to run.

The interviewer watches whether you notice the hard deadline (a suggestion that arrives after you've typed on is useless), size the model work, and decide what goes into the limited prompt. The walkthrough follows the 8 steps of the interview framework: scope, data, API, the busy path, numbers, services, scale and deep dives.

1. Scope: what version 1 must do

• Suggest the next few lines in gray as the developer types; Tab accepts, typing on ignores it.
• Use the open file and related code from the rest of the repository.
• Show the suggestion within about 300 ms of a pause in typing.

Park the rest: a chat panel for questions about the code (a later extra, with its own model), bots that review pull requests, and agents that run commands and edit many files. Each is a product of its own.

2. Data: three homes

• The editor session: the open file, the cursor position, recent edits, the other open tabs. It changes with every key, so it lives in the plugin and travels with each request.
• The repo index: built ahead, once per repository. The code is cut into chunks and indexed by meaning (embeddings and vector search, the same machinery as a support bot with RAG), next to a table of symbols: every function and class name and where it is defined.
• The usage log: whether each suggestion was accepted and how long it took to show. The acceptance rate is the main quality number, latency the other. The log keeps counts and timings, not code.

3. The API: one call per pause

Ana's cursor sits after total = in stats.py. When she pauses, the plugin sends:

POST /completions
{"path": "stats.py",
 "prefix": "from utils import safe_div\n\ndef mean(xs):\n    total = ",
 "suffix": "\n    return safe_div(total, len(xs))",
 "snippets": ["def safe_div(a, b): …"]}
  → 200 OK   {"text": "sum(xs)"}

Send the code before the cursor (the prefix) and after it (the suffix). The line below the cursor, return safe_div(total, len(xs)), is what tells the model that total must be a sum; code completion models are commonly trained for this fill-in-the-middle task. The snippets carry code from other files, like the definition of safe_div. The whole repository is too much (millions of tokens, while the model reads a few thousand), and the current line alone leaves the model guessing. The reply is short, so it can come back whole. Errors: 401 for a bad sign-in token, 429 past the rate limit.

4. The busy path: a pause, a request, gray text

1. The plugin notices the pause and sends POST /completions.
2. The server adds related code from the repo and builds the prompt.
3. A model server reads the prompt.
4. It writes sum(xs), one token at a time.
5. The plugin shows it in gray, and Tab accepts it.

It runs on every pause, and all of it must fit in about 300 ms.

5. The numbers

From the interviewer: at busy hours, 1 million developers are typing at once, and their pauses send about 10 requests a minute each.

• Requests: 1 million × 10 = 10 million a minute, ÷ 60 ≈ 170,000 a second (166,667).
• Model work: models read and write tokens, pieces of text of about 4 characters. Say a prompt is about 2,000 tokens and a suggestion about 50: 170,000 × 2,000 = 340 million tokens a second to read, and 8.5 million to write.
• The budget: about 300 ms from the pause to gray text, for every one of those requests.

All the model work runs on GPUs, the costliest machines in the system, so every request you avoid saves 2,000 tokens of work.

6. Services: gateway, context builder, model servers

• Gateway: checks the sign-in token and counts each developer's requests against a limit. Light work.
• Context builder: searches the repo index for related code, chooses what fits in the prompt and builds it.
• Model servers: run the model on GPUs, each working on a batch of requests side by side. The heavy part: you need far more of these than of anything else.

7. Scale: a cache for repeated prompts

Developers often send the same prompt again: they delete a word and type it back, undo, or pause twice at the same spot. Put a cache of recent suggestions beside the model servers, keyed by a hash (a short fingerprint) of the whole prompt plus the user, kept a few minutes. The context builder checks it first and calls the model only on a miss (cache-aside).

Don't key it by the line being typed: total = starts lines in millions of files, and a shared cache would show one company's code to another. More model servers would just redo the same work. The plugin saves requests too: when the developer types the very letters the gray text shows, it keeps showing the rest.

8. Deep dive: the latency budget

Out of 300 ms, the network takes about 50 ms there and back to a nearby data center (say; across an ocean it would be about 150 ms, half the budget, so model servers run in regions near developers), the queue about 50 ms at busy hours (say), and drawing the gray text about 10 ms. That leaves 190 ms for the model.

A model reads the whole prompt in one pass, its tokens side by side, then writes its answer one token at a time. Say a small model reads 40,000 tokens a second and writes 500, and a big one reads 4,000 and writes 50:

part                  small model   big model
network, both ways    50 ms         50 ms
queue                 50 ms         50 ms
read 2,000 tokens     50 ms         500 ms
write 50 tokens       100 ms        1,000 ms
draw the gray text    10 ms         10 ms
total                 260 ms        1,610 ms

The small model fits; the big one takes about 1.6 s, five times the budget. More GPUs don't fix that: they let you serve more requests at once, but each request still writes its 50 tokens one after another, about 1 s at 50 a second. Throughput is not latency.

So use a small, fast model for completions and keep the big one for the chat panel later. A completion is a short guess, ignored if wrong, 170,000 times a second. A chat question is rarer (a few an hour, against about 600 pauses an hour) and harder; people wait a moment for a good answer and read it as it streams in (server-sent events).

Last, debouncing and canceling: the plugin sends a request only after about 100 ms without a key, and cancels one that new typing made useless, so the model server drops it:

time      what happens
0 ms      Ana types t
80 ms     o
160 ms    t
260 ms    100 ms of quiet: request 1 leaves for "tot"
330 ms    a: request 1 is canceled
410 ms    l
510 ms    100 ms of quiet: request 2 leaves for "total"
770 ms    the gray text shows

Without the wait, 5 keys would send 5 requests, most of them out of date before they finish.

9. Deep dive: choosing the context

The prompt is capped at 2,000 tokens, because reading time grows with the prompt. The code around the cursor goes in first, say 1,000 tokens. The rest of the room goes to snippets from the open tabs and the repo index, each scored by how related it is: by meaning (a vector search over the chunks) or by name (a symbol used at the cursor, like safe_div, whose definition the symbol table finds). Fill it greedily: best score first, add each snippet that still fits, skip one that doesn't.

snippet          tokens   score   result
utils.py         300      0.91    in, 700 left
test_stats.py    500      0.84    in, 200 left
old_stats.py     600      0.80    skipped
models.py        400      0.62    skipped
README.md        150      0.55    in, 50 left

The prompt ends at 1,950 tokens. Stopping at the first snippet that doesn't fit would leave 200 tokens empty; forcing old_stats.py in would make 2,400. Greedy isn't always optimal (it's a small knapsack problem), but it's fast, and it runs on every request.

The index must also stay fresh: a day-old index can't find the function written this morning. Give every file a content hash, a fingerprint of its bytes. The plugin sends each file's hash, and an indexer re-chunks only the files whose hash changed and recomputes their embeddings, seconds after a save. A nightly rebuild misses today's code; re-indexing everything on every save wastes the work. File-sync tools use the same idea: hash the content, move only what changed.

10. Deep dive: privacy and limits

For many companies their code is their most private data, and every request carries some of it. The rule: the code is used for the suggestion, then forgotten.

• Keep prompts in memory only while the request runs.
• Log acceptance and timing, never the code: logs are copied, kept for months and read by many people.
• Never send or index files the repo marks private, like .env files full of passwords and keys.
• Keep cache entries apart per user, for minutes only.
• Limit each developer's requests at the gateway (a token bucket), answering 429 past the limit, so a runaway script can't take GPU time from everyone.
• Train on prompts only if the customer opts in.

What breaks first

• GPU capacity at busy hours: the queue grows and eats the budget. Watch queue time, and skip a suggestion rather than show it late.
• A huge repository: indexing takes a while. Use open tabs until the index is ready.
• A runaway client: a buggy plugin that never debounces. Rate limits protect everyone else.
• Quality drift: a new model or context rule can quietly make suggestions worse. Watch the acceptance rate.

Summary: the order to follow

1. Scope: gray text as you type, from the open file and related code, within about 300 ms.
2. Data: the editor session, a repo index (chunks by meaning, symbols), a usage log without code.
3. API: POST /completions with the prefix, the suffix, the path and snippets.
4. Busy path: a pause, a request, the model, gray text.
5. Numbers: about 170,000 requests and 340 million prompt tokens a second.
6. Services: a gateway, a context builder, model servers.
7. Scale: a cache of suggestions keyed by a hash of the prompt.
8. Deep dives: 190 ms for the model, so a small, fast one, with debouncing and canceling; the best snippets that fit 2,000 tokens, from an index kept fresh by content hashes; code used, then forgotten.

The same shape fits most AI features: find the deadline, size the model work, pick the smallest model that meets it, and spend the prompt on what matters most.