Design the feed: pull, push and the hybrid

Part of the free System Design course on LogicWiz, in the chapter: Design a news feed. The interactive whiteboard exercises and simulations run in the app.

Week 14, Day 2: Three hundred lookups per tap

"Opening the home tab fired 300 lookups, one for every account Rohan follows. At the evening peak the app asked for five million of them every second."

Meera, reading the load test

Step 1: the simplest feed, and the number that kills it

Start with what Meera first imagined. When Rohan opens the feed, the server reads his follow list, fetches the latest posts of each account he follows, merges them by time, and returns the top 20. This is fan-out on read, also called pull: the work of building the feed happens when someone reads it.

Say a typical reader follows 300 accounts. That is above the average of 200 followers per author, because people who open the feed often tend to follow more. At 17,000 reads per second that is 17,000 x 300 = 5.1 million lookups per second. One database is comfortable at about 2,000 simple queries per second. Even at 10,000 per machine you would need about 510 machines, only to read.

Meera said what many candidates say: "Add an index." An index makes each lookup fast. It does not reduce the number of lookups, which is the problem.

There is a second cost. A feed read now waits for 300 calls, and the slowest one decides the answer. If 1 in 100 calls is slow, the chance that at least one of 300 is slow is 1 - 0.99^300, about 95%. Almost every read feels the slow call.

Set the slow share to 1% and the number of calls to 300. Watch how the chance of a slow read climbs, then try 5 calls to see why a short fan-out is safe.

Step 2: do the work when the post is written

Flip it. When someone posts, copy the post id into each follower's feed ahead of time. This is fan-out on write, or push. Each user gets a feed store: a list of post ids, newest first, cut to the newest 500. Reading becomes two cheap steps: fetch the list, then hydrate the top 20, meaning fetch the full posts in one batch. Reads cost about 17,000 list lookups per second from memory. Hydration is the larger number: 17,000 x 20 posts is about 340,000 post lookups per second. Those go in batches of 20, and a cache of recent posts answers most of them, so the post store sees a small share.

The cost moves to writes: about 35,000 inserts per second at peak. Two rules keep that safe.

• Never fan out inside the POST request. The post service saves the post, puts a PostCreated event on a queue, and returns 201 (Created). Fan-out workers do the copying afterwards. Otherwise posting gets slower as an author gains followers.
• Add the post to the author's own feed at once, so they see it immediately, even if followers wait a few seconds.

Workers need "who follows this author?", so the follow graph is stored twice: follows (by follower) and followers (by followee). Both are sharded by their first column. Assume 100 million accounts following about 200 each. That is 20 billion rows at roughly 24 bytes, about 480 GB per table and about 1 TB in total. The trade-off is two writes for every follow.

Step 3: what breaks at 20 million followers

Push has a flaw, and the arithmetic finds it. One creator with 20M followers means 20M inserts. At 50,000 per second that takes 400 seconds, and while it runs, ordinary posts on the same queue and workers wait behind it. Many of those followers also never open the app, so most of the work is wasted.

The first fix is to push only to active users, those who opened the app in the last 30 days. A returning inactive user gets a feed rebuilt on first open, then cached.

The second fix is the hybrid. Authors above a threshold, say 100K followers, are not pushed at all. When Rohan opens his feed, the feed service reads his cached list and also fetches the latest posts of the few big accounts he follows. It merges the two by time. A user follows only a handful of such accounts, and their posts are in cache because so many people read them.

A candidate in a mock said: "Pick one, push or pull." The answer is that each side wins where it is cheap. Push wins for ordinary authors, where the copy count is small and reads dominate. Pull wins for huge authors, where copies are enormous and many followers are inactive. The trade-off: a merge step on reads and a threshold to tune, in return for bounded write cost and fast reads.

The finished design, and one request each way

flowchart LR
  A["App"] --> L["Load balancer"]
  L --> P["Post service"]
  L --> F["Feed service"]
  P --> S[("Post store")]
  P -. "PostCreated" .-> Q["Queue"]
  Q -.-> W["Fan-out workers"]
  W --> G[("Follow graph")]
  W --> C[("Feed cache")]
  F --> C
  F --> S

Posting. The app uploads images to object storage, then calls POST /posts. The post service saves the post, publishes PostCreated and replies 201. A worker reads the event. For an ordinary author it pages through followers in the followers table and adds the post id to each active follower's list. For a big account it does nothing.

Reading. GET /feed reaches the feed service. It reads the user's list of ids, fetches recent posts of followed big accounts, merges them by time, hydrates the top 20 from the post store and returns them with a cursor. Deleted posts are dropped during hydration.

Now build it yourself.

📌 Summary: Pull makes 300 lookups per read, about 5 million per second. Push copies ids into capped per-user lists through a queue. A hybrid skips accounts above about 100K followers and merges them at read time.

What Meera learns next

The shape is right, but the hardest cases are still open: a post that millions request at once, the cost of a like counter, and ranking. Next: the deep dives, the failures and the follow-ups.

Check your understanding

1. With pull, a user follows 300 accounts and the feed is read 17,000 times per second at peak. A teammate says an index will fix it. What is the real problem?
2. The post service copies the post id into 200 followers' feeds before replying 201. As accounts grow, what goes wrong?
3. A creator has 20 million followers. Workers insert 50,000 feed entries per second. How long does one post take, and what is the sensible design response?
4. Why does the design store the follow graph in two tables, follows and followers?
5. A user who has not opened the app for three months taps it. They have no feed list, because we only push to active users. What should happen?