Movie Reviews Aggregator

The problem

"Design a movie reviews aggregator", a site like Rotten Tomatoes. Yours collects critics' reviews from 300 review sites and shows a score for each movie that moves as reviews come in.

It reuses the Web Crawler's pattern. What's new: a fixed list of sources checked on a timer, a parser per site that breaks when the site changes, matching reviews to movies whose titles repeat, and a score that stays right. Park ratings from your own users: a design of its own.

The numbers

From the interviewer: 300 sources and about 600 new reviews a day in all.

• Fetches: checking each source's list of latest reviews every 10 minutes is 300 × 144 = 43,200 checks a day; with the 600 review pages, about 44,000 fetches a day. Over 86,400 seconds that is about 1 fetch every 2 seconds. The Web Crawler needed about 400 a second.
• The seen set: 600 review URLs a day for 10 years is about 2 million; at about 100 bytes each, about 200 MB, a plain table.
• Raw pages: at about 100 KB a page, 600 pages are 60 MB a day, about 22 GB a year.

One small machine is plenty. Say so: the hard part here is getting each review right, not fetching fast.

Schedules: a check time per source

Sources post at very different rates. Say 30 big outlets post about 500 of the 600 reviews a day, and the other 270 sites about 100 between them, one every few days each:

sources           reviews a day   check every
30 big outlets    about 500       10 min
270 quiet sites   about 100       6 h

Keep a sources table: list page (or feed), check interval, next check, parser. Big outlets every 10 minutes and quiet sites every 6 hours is 30 × 144 + 270 × 4 = 5,400 checks a day, 8× fewer than 43,200, with the big outlets just as fresh. Let it adapt: a check that finds new reviews brings the next one forward, an empty one pushes it back, up to a cap.

Checking every source every minute would be 432,000 checks a day, 720 for each review: the fastest way to get blocked. A nightly crawl of each whole site would show reviews up to a day late.

Parsers: one per site, each watched

Every site lays out a review its own way, so each source has a parser: where the critic, the movie's title, year and director, and the rating sit on its page. It also turns the site's scale into one out of 10: 3 of 4 stars is 7.5, 1.5 of 5 stars is 3.0.

A redesign rarely shows up as an error: the page still loads with 200 OK, and the parser quietly finds nothing. So watch each parser's success rate, the share of its pages where it found critic, movie and rating:

parser          Mon     Tue
Site A          100%    100%
Site C          100%    25%
all 300 sites   98%     97.5%

Site C posts about 4 reviews a day, and on Tuesday only 1 of 4 parsed, yet the total moves only from 98% to 97.5%. Alert on each parser against its own normal, as in Metrics and Alerting. Since the raw pages were kept, the fixed parser reads them again and no review is lost.

Matching: title, year and director

Titles repeat: David Lynch's Dune came out in 1984 and Denis Villeneuve's in 2021. Normalize the title (lowercase, no punctuation), find the catalog movies with that title through an index of titles, as in Search Engine, and keep the one whose year and director agree:

the review says                    match
Dune, 2021, Denis Villeneuve       Dune (2021)
DUNE (1984), David Lynch           Dune (1984)
Dune, Villeneuve, no year          Dune (2021)
Dune, 1984, no director            Dune (1984)
Dune, 1984, Denis Villeneuve       ask a person
Dune, no year, no director         ask a person

One clear match: attach the review. A conflict or too little to go on: a queue for a person, never a guess that puts a 1984 review into the 2021 score.

The score: a running sum and count

Keep a sum of ratings and a count in each movie's row; the score is the sum divided by the count. Say Dune (2021) has 9 reviews averaging 8.0, a sum of 72, and a 10th rates it 3.0: the sum becomes 75 and the score 75 ÷ 10 = 7.5. That's one small write per review, about 600 a day.

Averaging the old score with the new rating, (8.0 + 3.0) ÷ 2 = 5.5, gives one review the weight of the 9 before it. Averaging every review again on each view repeats the work: say a popular movie has 300 reviews and its page opens 500 times a second at busy hours, that's 150,000 ratings read a second for a score that changes a few times a day.

Key each review by its URL, so a review fetched twice counts once, and when a re-parse changes a rating, take the old one out before adding the new.

Same pattern as the Web Crawler

The fetching is the crawler's: a queue of work for the fetcher, a gap per site, robots.txt, retries with growing waits, a seen set, raw pages in storage. What doesn't carry over is the scale: no link following, no split frontier, no Bloom filter.

Summary

• Scope: 300 known sources; parse, match and score their reviews.
• Data: sources, movies by title, year and director, reviews keyed by URL.
• API: GET /movies/{id} returns the score and its reviews.
• Numbers: about 1 fetch every 2 seconds, one machine.
• Services: a fetcher on a schedule per source, parsers, a matcher.
• Parsers: one per site, each with a success rate and an alert.
• Matching: title + year + director, a person for the rest.
• Score: a running sum and count per movie.