Recommendations (Netflix)

The problem

"Design the recommendations on Netflix's home page": the rows like "Because you watched Orbit" that are different for every member. The page must appear at once, reflect what you watched a minute ago, work for someone who joined today, and the team must be able to prove that a new model is better than the old one.

This practice problem reuses the pattern of Design Autocomplete: rank ahead in a batch job, then serve ready lists from memory. What's new is that the lists are per member, built from what people watched, with a light touch at request time, a cold start, and an A/B test. Park artwork chosen per member and search: each is a design of its own.

The numbers

From the interviewer: 200 million members, and about 100 million open the app each day, about 3 times each. Round a day to 100,000 seconds (86,400 really); busy hours run about 5× the average.

• Home pages: 100 million × 3 = 300 million a day, 3,000 a second on average (about 3,500 with 86,400), so about 15,000 a second at busy hours.
• Ready rows: say 20 rows of 20 titles per member, 400 ids of 8 bytes: 3.2 KB a member, 640 GB for 200 million members. That fits in the memory of about 10 servers of 64 GB.
• Pairs of titles: say the catalog has 10,000 titles: 10,000 × 9,999 ÷ 2 is about 50 million pairs. Keeping the top 100 for each title leaves 1 million rows.

Co-watching: people who watched X also watched Y

Ana has been watching the series Orbit, and her row "Because you watched Orbit" needs titles. Take a toy history of 7 other members:

member   watched
ben      Orbit, Comet, Baker
cy       Orbit, Comet
dee      Orbit, Comet, Noir
eve      Orbit, Baker
fay      Baker, Coast
gus      Baker, Coast
hal      Baker

Four people watched Orbit. Among them, 3 also watched Comet, 2 watched Baker and 1 watched Noir. That's the co-watch count: for each pair of titles, how many people watched both. Comet goes first.

Two tempting answers are wrong. Baker is the most watched title overall (5 of the 7), but plain popularity would put the same hit first in every row, whatever you watched; that belongs in a "Popular" row. Noir looks perfect, since everyone who watched it also watched Orbit, but that's one person. A common refinement divides each count by how popular the titles are and drops pairs with too few viewers.

Rank ahead, re-rank lightly

At 15,000 home pages a second, counting over billions of views on each request is out of the question. Split the work:

• The nightly job: count co-watching for every pair of titles over everyone's history; build each member's rows from the titles they watched, minus what they've seen; write one "Popular in your country" row per country. It saves each member's ready rows in a key-value store in memory, keyed by member id.
• When Ana opens the app: GET /home is one read of her rows. The home service then makes a light re-rank of the few hundred titles in them: the Orbit row moves up because she finished it 5 minutes ago, Orbit itself is hidden, and at 11 pm on a weeknight shorter titles come first. That's a few milliseconds, not a recount.

Cold start: new members and new titles

Ben joined a minute ago, and a new title, Nova, came out today. Co-watch counts know nothing about either. Ben's page starts with what many people like: the "Popular in your country" row (in the toy history: Baker, Orbit, Comet) and picks from editors. Many services also ask a new member for a few titles they like. After a few views, the next nightly run gives him rows of his own.

Nova needs its first viewers before it can have counts, so editors place it in a "New" or picked row, and it can sit next to titles of the same genre. An empty page would lose Ben in his first minute, and a random page from a 10,000-title catalog would look broken.

The A/B test: does a new model help?

The team trains a new model for the rows, and "helps" means more watch time. Split members at random: a hash of the member id puts each one in group A (the old model) or B (the new one), the same group on every visit, and the nightly job writes each member's rows with their group's model. After a few weeks, compare watch time per member; the gap must be bigger than the day-to-day noise.

Comparing this month with last month mixes the model with the season: holidays, a hit series. Comparing two countries mixes it with taste and catalog. And measure watch time, not clicks: a flashy row can win clicks and lose viewing.

Same pattern as Autocomplete

The shape is Autocomplete's: count from the logs in a batch, write the answers ready, serve them from memory, and accept answers a few hours old because tastes move slowly. The key changes from a prefix to a member, so there are 200 million ready answers, and the request adds a light re-rank that the night couldn't do.

Summary

• Scope: rows for each member, a touch at request time, cold start, a test.
• Data: viewing history → co-watch counts → ready rows per member.
• API: GET /home returns the member's rows, re-ranked.
• Numbers: about 15,000 home pages a second at busy hours; 640 GB of ready rows.
• Services: a nightly job writes the rows; the home service re-ranks.
• Cold start: popular in your country, and editors' picks.
• A/B test: random halves by member id, compare watch time.