Content Moderation

The problem

"Design content moderation for a social app." Every post, its images and its text, is checked against the app's rules. A big model can judge most posts well, but it runs on costly GPUs; people judge better still, but they are slow and few. The design sends each post to the cheapest check that can settle it.

This practice problem reuses Design LeetCode's job queue, where work waits until a worker is free, and ChatGPT's model calls, a big model on GPU servers fed by a queue. What's new is judgment: some posts need a person, users report posts, authors appeal, and every post is either shown or held while it waits. Park live video and chats about removals for later.

The numbers

From the interviewer: 10 million posts a day, images and text. Round a day to 100,000 seconds (86,400 really): that is 100 posts a second on average, 500 at busy hours, about 5× (about 580 with 86,400).

Say the big model needs about 1 second of one GPU per post. Sending every post through it would keep about 500 GPUs busy at busy hours, mostly judging posts that are plainly fine.

Fast checks first

Two checks are cheap enough to run on every post before it is shown:

• A list of known banned images. Each image already removed is kept as a hash of how it looks, so a resized copy still matches, and checking a new image is a lookup in microseconds. Companies also share such lists for the most serious kinds of content.
• A small text classifier on a CPU, a few milliseconds a post, that tells plainly fine text from plainly bad.

Say these settle 9 posts in 10. Only the 1 in 10 they can't settle goes on the check queue to the big model: 50 posts a second at busy hours, so about 50 GPUs instead of 500. Dropping the big model doesn't work either: a small classifier misses what takes context, such as a joke that reads as a threat or text inside an image, and everything it's unsure about would go to people, 1 million posts a day.

Show at once, or hold

A post on its way to the big model waits a second or more, and one that ends up with a person can wait hours. Whether it is visible meanwhile is a decision by risk:

• Show at once, check right after: trusted accounts, such as one with 3 years of posts and no removals, or a long-time member of a small group. A rare removal comes minutes later.
• Hold until checked: an account made 10 minutes ago, an account with recent removals, a post the small classifier scores as a likely scam, and paid ads that reach strangers.

Most posts are shown at once, so most people never wait.

People for the unsure middle

Say the big model is unsure about 1 post in 20 it sees: about 50,000 posts a day go to a review queue, and reviewers take the next one when free. If a reviewer handles about 1,000 a day, that is work for about 50 reviewers, before reports and appeals.

The queue is ordered by priority, not by arrival. A simple rule ranks by the chance a post breaks a rule times its reach, the people who see it in an hour while it waits. Neither alone works: by reach, the maybe-spam would come second; by chance, the barely seen spam first.

post                              chance    reach    priority
shown, likely a scam              0.8       10,000   8,000
shown, maybe harassment           0.5       4,000    2,000
shown, maybe spam                 0.2       5,000    1,000
shown, likely spam                0.9       100      90
held, new account, likely spam    0.7       0        0

A held post reaches no one, so it can wait, though not forever: its author is waiting too. Real queues also weigh how serious the rule is, so a threat outranks spam at the same reach. And every reviewer's decision is a labeled example: the models are retrained on them, and the unsure middle shrinks.

Reports

Users report posts the models let through. Reports go into the same review queue, with one item per post: 300 reports in an hour are one review, not 300, and each report raises its priority. Removing a post automatically after, say, 100 reports would let any group take down a post it dislikes by reporting together; a report is a signal for a reviewer, not a verdict. A separate reports team with its own queue would review some posts twice.

Appeals

An author who thinks a removal was a mistake appeals. The appeal goes into the same queue for anyone but the first reviewer: people tend to stand by their own calls, and running the same model again gives the same answer. If the second reviewer disagrees, the post comes back, and the reversal is a label too: it shows where the model or the rule is unclear.

Same pattern as LeetCode and ChatGPT

From LeetCode: work waits in a queue, and workers, here the model servers and the reviewers, take the next item when free, so busy hours lengthen the queue instead of losing posts. From ChatGPT: the big model runs on GPU servers fed by a queue, and it is the costly part, so the design spends it only where cheaper checks can't decide.

Summary

• Scope: every post checked, images and text; reports and appeals.
• Numbers: 500 posts a second at busy hours; 50 GPUs instead of 500.
• Scale: a hash list and a small classifier on every post, the big model for the 1 in 10 they can't settle.
• Show or hold: by risk; trusted accounts shown at once and checked right after.
• Review: a queue ranked by chance × reach; reviewers' calls retrain the model.
• Reports: into the same queue, one review per post.
• Appeals: a second reviewer, never the first.