FjellSystem Design Lab
Lesson 10.2• 11 min read

Fan-out on write, on read, or both

Push versus pull, the cost of fanning out a celebrity’s tweet, and the hybrid that merges highly followed accounts at read time.

What you will learn

  • Compare fan-out on write (push) with fan-out on read (pull).
  • Explain why highly followed accounts break push.
  • Design the hybrid, and choose where to draw its threshold.

Two ways to build a home timeline

  • Fan-out on write (push): when someone tweets, copy the tweet ID into the cached home timeline of every follower. Reading a timeline is then a single list read.
  • Fan-out on read (pull): store each tweet once. When someone opens their home timeline, read the recent tweets of every account they follow and merge them.

The primer's design pushes. Fanning out to followers runs at 60,000 deliveries per second, which "will overload a traditional relational database". So the home timelines live in a store with fast writes, a memory cache or NoSQL database, while each user's own tweets live in SQL.

When a user posts, the Fan Out Service:

  1. asks the User Graph Service for the user's followers (held in the memory cache);
  2. stores the tweet in each follower's home timeline in the memory cache, an O(n)O(n) operation: "1,000 followers = 1,000 lookups and inserts";
  3. adds the tweet to the Search Index Service, stores media in the Object Store, and queues push notifications.

Reading the home timeline then takes one O(1)O(1) fetch of tweet and user IDs from the cache, plus two multigets for the tweet and user details.

Push Pull
Work per tweet One insert per follower One insert
Work per timeline read One list read One read per followed account, then a merge
Memory A copy of each tweet ID per follower Each tweet stored once
Breaks down when An author has millions of followers A reader follows many accounts, or reads very often

The celebrity problem

Push costs one insert per follower. The primer warns that users with millions of followers "could take several minutes to have their tweets go through the fanout process". It adds that this "could lead to race conditions with @replies to the tweet": a reply can arrive before the tweet it answers. You can mitigate that by re-ordering tweets at serve time.

Twitter's own figures (2013) show the scale: a median of 3.5 seconds to deliver a tweet to 1 million followers, and up to 5 minutes at the 99th percentile.

The hybrid

The primer's fix: "avoid fanning out tweets from highly-followed users". Instead, find their recent tweets when the timeline is read, merge them with the user's precomputed home timeline, then re-order at serve time. Twitter did the same: "For people like Taylor Swift don't bother with fanout anymore, instead merge in her timeline at read time."

  1. Most authors: fan out on write, usually asynchronously through a queue.
  2. Highly followed authors: skip the fan-out and store the tweet once.
  3. On read: fetch the precomputed timeline, then fetch the recent tweets of the few highly followed accounts the user follows, and merge by time.

Where to draw the line is a design choice, so tune it against measurements. A lower threshold means fewer writes but more merging on every read.

The primer adds more savings:

  • Keep only several hundred tweets per home timeline in the cache.
  • Keep timelines only for active users. For someone inactive for 30 days, rebuild on demand: ask the User Graph Service who they follow, read those tweets from the SQL database and fill the cache.

Push, pull or hybrid fan-out

What one tweet costs to deliver, and what one home timeline costs to read.

Strategy
Timeline inserts for this tweet
25,000
Time to reach every follower
88 ms
Reads per home timeline (reader follows 300)
4

This author is below the threshold, so the tweet is pushed. The reader also merges 3 followed celebrities at read time. Delivery speed assumes about 285,714 inserts per second, from Twitter’s reported median of 3.5 s to reach 1 million followers.

Figure 2: Hybrid timeline fan-out architecture routing regular posts to async push workers while deferring celebrity reads to serve-time merging.
  • Tweet Author (Post): 1 × Clients & Users
  • Write Load Balancer: 2 × Application Load Balancer
  • Fanout Decision Router: 4 × App Compute Server
  • Fanout Queue (Regular Users): 3 × Message Queue / Kafka
  • Celebrity Recent Tweets: 2 × Primary Relational DB
  • Fanout Worker Pool: 8 × App Compute Server
  • Redis Follower Feed Inboxes: 4 × Redis In-Memory Cache
  • Tweet Author (Post) sends requests to Write Load Balancer
  • Write Load Balancer sends requests to Fanout Decision Router
  • Fanout Decision Router sends requests to Fanout Queue (Regular Users)
  • Fanout Decision Router sends requests to Celebrity Recent Tweets
  • Fanout Queue (Regular Users) sends requests to Fanout Worker Pool
  • Fanout Worker Pool sends requests to Redis Follower Feed Inboxes
  • Celebrity Recent Tweets sends requests to Redis Follower Feed Inboxes

Key takeaways

  • Push copies each tweet ID into every follower's cached timeline, so a read is one list fetch; pull stores each tweet once and merges at read time.
  • Push costs one insert per follower: 60,000 deliveries a second would overload a relational database, and an account with millions of followers can take minutes to reach them all.
  • The hybrid pushes most tweets, stores highly followed accounts' tweets once, and merges those in when a timeline is read.
  • A lower threshold means fewer writes but more merging on every read; tune it against measurements, and keep timelines only for active users.

Sources

Adapted from The System Design Primer by Donne Martin, licensed under CC BY 4.0. Changes were made.