Translation Service

The problem

"Design a translation service like Google Translate." People paste text on a website, and thousands of apps call an API to translate chat messages, product pages and their own buttons. A model on GPU servers does the translating. Version 1 handles text and documents; park speech and the words in photos.

This practice problem reuses the pattern of Design ChatGPT. What's new: whole documents next to short texts, repeats worth caching, a source language nobody named, customers' brand names, and users' corrections.

The numbers

From the interviewer: 1 billion requests a day, most of them a sentence or a short string. Round a day to 100,000 seconds (86,400 really); busy hours run about 5× the average.

• Requests: 1 billion ÷ 100,000 = 10,000 a second (about 11,600 with 86,400), so 50,000 a second at busy hours.
• Model servers: say one server translates about 500 short texts a second, batching them: 50,000 ÷ 500 = 100 model servers, plus spares. Forgetting busy hours gives 20.

Two paths: short texts and documents

POST /translate   {"text": "Obrigado!", "from": "auto", "to": "en"}
  → 200 OK   {"text": "Thank you!", "detected": "pt"}

POST /documents   contract.pdf, "to": "es"
  → 202 Accepted   {"job": "d81"}
GET /documents/d81
  → 200 OK   {"status": "done", "file": "…/d81-es.pdf"}

A 300-page contract is about 10,000 sentences. At 500 sentences a second that's 20 seconds of a whole model server: far too long for one request, and the short texts behind it would wait. So there are two paths. The fast path answers a chat message, a search box or an app's button in a fraction of a second. The document path stores the file in object storage, answers 202 Accepted with a job id, and splits the file into sentences on a doc queue. Model servers take live requests first and document sentences when there's room. The docs service puts the translated sentences back into the file's layout, and the customer polls or gets a webhook.

Caching repeated translations

Unlike chat answers, translations repeat: apps send Sign in and Add to cart every day. Put a cache in front of the model, keyed by a hash of the text, the source language and the target language; the hash keeps keys short however long the text. Say 60% of requests hit: the model sees 20,000 requests a second instead of 50,000, so 40 model servers instead of 100.

The key must name both languages: "Gift" is English for a present and German for poison, so a key of the text alone could hand a German-to-English request the English-to-German answer. Keying by user would store Sign in once per user and almost never hit. Say 100 million cached entries of about 500 bytes: 50 GB, a few in-memory servers. When a new model ships, put its version in the key too, so old answers age out.

Detecting the source language

Most people never pick the source language, so the app sends "from": "auto". A small, fast language detection model guesses it first, in about a millisecond on an ordinary CPU: telling languages apart is far easier than translating. The reply says what it found, and the app lets the user change it, since a very short text such as "Gift" fits several languages. Translating from all 100 languages would multiply the GPU work by 100, and the user's phone language fails for a German phone pasting a Japanese menu. A design may run one model per language pair or one model for many languages; either way the request needs its pair first.

Glossaries per customer

A customer, Nimbus, sells a product called Cloud Drive that must stay in English, but its German screens read Wolkenlaufwerk. Give each customer a glossary of terms to keep. A common way to apply it: swap each term for a placeholder the model passes through untouched, then swap it back. The glossary's id joins the cache key (text + from + to + glossary), or customers get each other's cached answers. A model trained per customer would cost weeks of GPU time for a list of names, and one shared do-not-translate list fails when another customer sells a product called Spring.

Corrections as quality signals

POST /corrections   {"text": "Gift", "from": "de", "to": "en",
                     "model": "v7", "suggested": "poison"}
  → 202 Accepted   {"status": "for review"}

A Suggest an edit button collects users' fixes. One person's edit is a hint, not a fact; many people making the same edit is a strong signal. Store each edit with its text, languages and model version, and let reviewed edits become training data for the next model. Never put an edit straight into the cache, where one joker's version would reach millions, and never retrain on raw edits, which would teach the model their jokes and spam.

Same pattern as ChatGPT

The serving side carries over: GPU model servers are the scarce, costly part, they take work from a queue in batches, and each customer's key has a limit. What changes is the work. Translations are short and often repeat, so a cache pays off as it doesn't for chat; documents need a background path, as any long job does; and each request first needs its language and its customer's glossary.

Summary

• Scope: short texts and documents; speech and photos later.
• Numbers: 50,000 requests a second at busy hours, about 100 model servers.
• Two paths: a fast API, and documents as jobs (202 and a job id) on a low-priority queue.
• Cache: a hash of text + from + to; 60% hits leave 40 model servers.
• Language: a small, fast model detects it first, and the reply says what it found.
• Glossaries: each customer's terms kept, and in the cache key.
• Corrections: signals for the next model, reviewed, never shown as fact.