Support Bot With RAG

What the interviewer is asking

"Design a support bot that answers from our docs" is a common AI system design question: a chat box on a help center, backed by a large language model (LLM), that must answer from the company's own pages, not from whatever the model remembers. The pattern is retrieval-augmented generation (RAG): find the few passages that match the question, then let the model write the answer from them. The interviewer wants to see how documents become searchable by meaning, what the numbers say about cost, and how you keep the bot from inventing answers. The walkthrough follows the 8 steps of the interview framework: scope, data, API, the busy path, numbers, services, scale and deep dives.

1. Scope: what version 1 must do

• Answer questions typed into a chat box, from the company's help center and docs: 10,000 pages.
• Cite its sources: each answer shows the pages it came from.
• Say "I don't know" and offer a human when the docs don't cover the question.
• Stay fresh: the docs change every day, and the answers must follow.

Name the rest and park it: acting on the customer's account (refunds, plan changes) needs permissions and an audit trail; voice and other languages wrap more around the same bot.

2. Data: pages, chunks and embeddings

A customer types "I can't log in". The page that helps is called "Reset your password", and the two share no words, so a keyword index alone would miss it. The bot has to find text by meaning:

• Page: its URL, title and text, plus a content hash for the daily update.
• Chunk: a passage of about 500 words, with the page it came from, so an answer can cite it.
• Embedding: for each chunk, a list of about 1,000 numbers that places text by meaning, so similar texts get nearby lists.
• Conversation: the messages and the sources each answer used, for follow-up questions and review.

Past questions with the answers agents gave only cover questions asked before, and go stale as the docs change. The docs are the source of truth.

3. The API: one call, a streamed answer

POST /support/chat
{"message": "I can't log in"}
  → 200 OK, streamed
"Open Security settings, …"
"…then choose Reset password."
{"sources": [
 {"title": "Reset your password",
  "url": "example.com/reset"}]}

The model writes a few words at a time, and a whole answer takes several seconds, so the reply streams: the first words show up in about a second over one response that stays open (server-sent events are a common choice), and the sources come last. Sending back the 5 best passages instead would turn the bot into a search box.

4. The busy path: one question

1. The support API gets the message.
2. It embeds the question with the same model used for the chunks.
3. It finds the 5 chunks with the nearest embeddings, among the docs this customer may see (their sign-in says who is asking).
4. It sends the LLM the rules, the 5 chunks as numbered sources, and the question.
5. The LLM writes an answer from them, citing its sources.
6. The API streams the answer back, then the sources as titles and links.

5. The numbers

From the interviewer: 10,000 pages, about 3 chunks each, so 30,000 chunks; embeddings of about 1,000 numbers, 4 bytes each; 100,000 questions a day.

• Index: 30,000 × 1,000 = 30 million numbers × 4 bytes = 120 MB, plus about 90 MB of text (30,000 × 3 KB). Say one server has 100 GB of memory: it fits many times over, so there is nothing to shard. A second copy covers a failed server.
• Questions: 100,000 a day is about 1 a second (a day ≈ 100,000 seconds), about 5 a second at busy hours. One app server answers about 1,000 requests a second.
• Tokens: a token is a word or a piece of one; 500 words are about 600 tokens. A prompt with the question and 5 chunks is about 3,000 tokens, an answer about 300: 100,000 × 3,000 = 300 million tokens read a day, 30 million written.

The cost is the model's tokens, and it grows with every chunk in the prompt. That's why the bot sends 5 chunks, not 50.

6. Services: ingestion and the support API

• Ingestion jobs: split a changed page into chunks, embed every chunk, save the vectors with their text and source page.
• Support API: embed the question, find the nearest chunks, send the prompt to the LLM service, stream the answer.

The docs live as files in object storage. Each day, every changed page becomes a job on a queue, so a slow or failed page is retried without touching answers. Both sides must use the same embedding model: an embedding from one model means nothing to another, so a new model means embedding every chunk again.

7. Scale: plan for the model

Say the model writes about 50 tokens a second: a 300-token answer streams for about 6 seconds. At 5 questions a second, 5 × 6 = 30 answers are being written at once. Holding 30 open streams is easy for one API server. The model is what to plan for: a hosted model's limits must allow 30 at once, or your own GPU servers batch them together. Popular questions repeat, so a cache of their answers, emptied at each docs update, saves model calls.

8. Deep dive: why not just ask the model?

• The model alone answers from its training, which never saw these help pages as they are today. When it doesn't know, it may still write a confident answer that sounds right: a hallucination.
• All the docs in the prompt: 30,000 chunks × 600 tokens = 18 million tokens with every question. Say the model reads at most 200,000 tokens at once: 90 times too much. Even if it fit, each question would cost 6,000 times the tokens.
• Retrieving 5 chunks: about 3,000 tokens, all about the question.

Training the model on the docs (fine-tuning) is another common follow-up: it would have to be redone with every daily update, and it can't point to the page an answer came from.

9. Deep dive: ingestion

Chunks overlap a little, so steps that cross a border stay together in one chunk. A tiny page of 6 sentences, in chunks of 3 that overlap by 1:

chunk   sentences
1       1–3
2       3–5
3       5–6

About 500 words is a balance: small enough to be about one thing (one embedding for a whole page is a blur of its topics), big enough to make sense alone ("It works for 1 hour." means little by itself).

Most pages don't change on a given day. Each page's content hash, a short fingerprint of its text, is stored with its chunks; the daily job hashes every page again, which needs no model, and only pages whose hash changed, or that are new, are split and embedded again.

page (chunks)   old → new     re-embedded
Passwords (3)   c102 → c102   0
Invoices (2)    c527 → e85e   2
Plans (4)       ba35 → ba35   0
Accounts (2)    0eb5 → 0eb5   0
Two-step (3)    e0dd → ff18   3
Pause plan (2)  – → d315      2

That's 7 of 16 chunks. At full size, say 1% of the 10,000 pages change on a day: about 100 pages, 300 chunks instead of 30,000.

10. Deep dive: vector search

Closeness is usually measured with cosine similarity: how closely two embeddings point the same way, 1 for the same direction, 0 at right angles. With toy embeddings of 2 numbers, for Q = [2, 5], "I can't log in":

chunk                embedding   cosine
B Account locked     [3, 5]      0.99
A Reset password     [1, 6]      0.98
E Change email       [4, 4]      0.92
D Can't get invoice  [5, 2]      0.69
C Update your card   [6, 1]      0.52

B and A win with no word in common with the question, while D, which shares "can't", scores low. Comparing the question with all 30,000 chunks is 30 million multiply-adds, tens of milliseconds. At many millions of chunks, an approximate nearest-neighbor index (such as HNSW) checks only a small part and still finds nearly the best.

"What does E1047 mean?" brings back chunks about other error codes, since codes that differ by a digit mean nearly the same thing to an embedding. A keyword index finds the exact term. Hybrid search runs both and merges the two ranked lists, for example with reciprocal rank fusion (each list adds 1 ÷ (60 + rank)), so a chunk high on either list reaches the top 5.

11. Deep dive: grounded answers

Answer the customer's question.
Use only the sources below. Cite them as [1], [2].
If they don't answer it, say you don't know.

Sources:
[1] Reset your password: Forgot your…
[2] Account locked: After 5 failed…
…
Question: I can't log in

That rule makes the answer grounded: written from the chunks, not from memory, each claim pointing to a source. Without it, the model fills gaps from memory, such as a refund window that sounds right and isn't this company's.

When the docs don't cover a question, even its nearest chunks are loosely related. Say below a best score of 0.75, the bot says it doesn't know and offers a human:

question              score   bot
Change my email?      0.90    answers
Pay by invoice?       0.86    answers
Student discount?     0.62    a human
Charged twice?        0.81    answers
Data kept in the EU?  0.58    a human

The right threshold depends on the embedding model, so pick it with a test set: a few hundred real questions with known good answers, run again whenever the prompt, the model or the threshold changes.

What breaks first

• The model: 30 answers at once at busy hours; watch its limits, cache popular answers, keep prompts small.
• Stale answers: a page that failed to re-ingest; retry it from the queue.
• Exact terms: error codes and product names; hybrid search.
• Private docs: filter chunks by what the signed-in customer may see, before they reach the prompt.

Summary: the order to follow

1. Scope: answers from 10,000 help pages, with sources, or "I don't know" and a human.
2. Data: pages, chunks of about 500 words with their page, an embedding for each, conversations.
3. API: POST /support/chat, the answer streamed, then its sources.
4. Busy path: embed the question, find the 5 nearest chunks, the LLM answers from them.
5. Numbers: a 120 MB index in memory, about 3,000 tokens a question.
6. Services: ingestion jobs fed by a queue; the support API.
7. Scale: 30 answers at once: plan for the model.
8. Deep dives: why RAG; overlapping chunks, re-embedding only changed pages; cosine similarity plus keywords; grounded answers with a threshold.

The same steps work for any "design X" question.