LinkedIn Feed

The problem

"Design the LinkedIn feed." You see posts from your connections, what they like and comment on, and posts from the companies and influencers you follow. Park messaging and job listings: each is a system of its own.

This practice problem reuses the pattern of Twitter at Scale. What's new: your network's activity is content too, so posts from people 2 steps away reach you through a connection; likes and comments far outnumber posts and must be grouped; and reading packs into work hours.

The numbers

From the interviewer: 100 million daily users, each opening the feed about 6 times a day, while only about 1 in 50 posts on a given day. People read at work: say nearly all of a day's opens land in about 8 hours, about 30,000 seconds (28,800 really).

• Opens: 100 million × 6 = 600 million a day ÷ 30,000 = 20,000 a second in work hours (about 21,000 with 28,800). Spread over the whole day's 100,000 seconds it would look like 6,000: the peak is about 3× the average.
• Posts: 100 million ÷ 50 = 2 million a day, about 70 a second, against Twitter's 1,000.
• Likes and comments: say 2 per user a day, 200 million: about 100× the posts.

Say you have about 500 connections: together they write about 10 posts a day, under 2 for each of your 6 opens. Your own network's posts can't fill the feed.

Second degree: how posts travel

Raj isn't your connection, but Ana is his, so he's 2 steps away. When Ana comments on Raj's public post, the comment is an activity event (who, what, which post), and it's pushed to Ana's connections just like a post. It carries Raj's post into your feed with a reason attached: "Ana commented". The push is bounded, because LinkedIn caps connections at 30,000.

The alternatives don't scale. Pulling the posts of everyone 2 steps away at every open means 500 × 500 = up to 250,000 people (fewer, since networks overlap), more than 1,000× Twitter's 200 follows. Pushing every post to everyone 2 steps from its author costs up to 250,000 writes per post, 500× a normal post, into the feeds of people who have never heard of Raj. A post must also allow it: one shared with connections only doesn't travel past them.

Activity: one item per post

Likes and comments run about 100× posts, and many land on the same few posts. In one hour, four of your connections might comment on Raj's post:

event            about
Ana commented    Raj's post
Ben commented    Raj's post
Cy commented     Raj's post
Dee commented    Raj's post
Ben liked        Mia's post

Group the events by the post they're about: one item per post, "Ana and 3 others commented", and "Ben liked this" for Mia's, updated as more events come in. Four items in a row would make the feed mostly repeats; keeping only the first would throw away that four of your connections cared, the best reason to rank the post high.

One post, two ways in

As with Twitter's celebrities, the companies and influencers you follow have millions of followers, so their posts are pulled at read time and merged in. Now an influencer's post can arrive twice: pulled because you follow her, and pushed because Ben commented on it.

Only the feed service holds both copies, at read time. So it gathers the ready items from the feed cache and the pulled posts, groups everything by post before ranking, and ranks the merged items. The fan-out workers can't do it, since the pulled copy only exists when you open the feed. Hiding repeats in the app is too late: the ranker has already spent two of your top 10 on one post, and you see 9 different posts instead of 10.

Busy mornings, quiet nights

Say one feed server handles about 1,000 opens a second. The morning needs about 20 feed servers; the day's average would need 6. Size the read side for the peak and add servers on a schedule before the work day, then remove them at night. Members span time zones, so in practice each region has its own morning and its own peak.

The quiet hours are free capacity. Building and ranking feeds, and pushing new posts, likes and comments, must happen during work hours, while people read. Retraining the ranking model on yesterday's clicks, counting views for authors' weekly stats and recomputing "people you may know" can wait: they become batch jobs that run at night.

Same pattern as Twitter at Scale

The engine is Twitter's: a queue of new posts and activity, fan-out workers pushing into a feed cache split by user id, and huge accounts pulled at read time and merged in. The companies and influencers play the celebrities. What changes is what gets pushed (activity, not only posts) and when the load comes.

Summary

• Scope: posts and activity from your network, people 2 steps away, companies and influencers.
• Data: activity events (who, what, which post) beside the posts.
• Numbers: about 20,000 opens a second at work, 3× the average; about 70 posts a second.
• Second degree: a connection's like or comment carries the post to you.
• Activity: one item per post, "Ana and 3 others commented".
• One item: pushed and pulled copies grouped by post, before ranking.
• Scale: feed servers scheduled for the morning; batch jobs at night.