Case Study 4: Instagram / Twitter News Feed

Show the posts of the people a user follows in a fast, personalised home timeline, in milliseconds, for hundreds of millions of users.

C.1 Requirements

Functional requirements:

• A user can create a post (with text, photo, video).
• A user can follow / unfollow other users.
• Home feed: the latest posts of followed users, ranked or in reverse-chronological order.
• Users can like, comment and share; every post shows its like count.
• The feed is an infinite scroll: cursor-based pagination.
• User profile timeline: all posts of one user.
• Notification of a new post to followers (optional, async).

Non-functional requirements:

• High availability 99.99%: a slightly stale feed is fine, but the feed must not go down (AP over CP).
• Feed load latency p99 < 200 ms, post publish < 500 ms.
• Eventual consistency is OK: it is fine if a new post shows up for followers within 5-10 seconds.
• Scale: 50 crore (500 million) DAU, a read-heavy system (read:write about 100:1).
• The like counter can be approximately correct, but never negative or badly wrong.
• Durable media storage (11 nines): an S3-like object store + CDN.

C.2 Estimation (back-of-the-envelope)

C.3 API design

C.4 Data model

C.5 High-level architecture

• The photo is first uploaded straight to S3 through a pre-signed URL from the Media Service; images load from the CDN.
• Post publish: LB -> API Gateway -> Post Service writes to Cassandra with a Snowflake ID, and an event goes on Kafka post-created.
• Fan-out Workers paginate the followers and ZADD the post_id into every active follower's Redis timeline feed:{id}; celebrities are skipped.
• Feed read: Feed Service merges the Redis IDs + celebrities' posts from Cassandra and gets them scored by the Ranking Service.
• Like Service INCRs the Redis Counters; the like event goes through Kafka and the Counter Aggregator flushes to Cassandra in batches.

C.6 Main flows

Post publish and fan-out on write:

1. The client first gets a pre-signed URL from /media/upload-url and uploads the photo directly to S3.
2. The client sends POST /v1/posts with an Idempotency-Key.
3. The API Gateway checks auth and rate limit; the request reaches Post Service.
4. Post Service creates a Snowflake post_id, writes to the Cassandra posts table and returns 201 to the client.
5. Post Service puts an event on the Kafka topic post-created (author_id, post_id, ts).
6. The Fan-out Worker reads the event; if the author is a celebrity it skips (pull model), otherwise it paginates the followers list.
7. The post_id is ZADDed into every active follower's Redis feed:{id} sorted set, trimmed at size 500.
8. Inactive followers (no login for 30 days) are skipped; their feed is rebuilt at login.

Home feed read (hybrid merge):

1. The client sends GET /v1/feed?cursor=.
2. Feed Service reads the top N post_ids of feed:{user_id} from Redis (after the cursor).
3. It picks the celebrities from the following list and pulls their latest posts from Cassandra/cache.
4. The two lists are merged and the Ranking Service scores them.
5. The post IDs are hydrated with a multi-get from the post cache (Redis); Cassandra on a miss.
6. Like counts from the Redis counters and the CDN media URLs are attached.
7. The response is returned with next_cursor; the client loads images from the CDN.

Liking and counter update:

1. The client sends POST /v1/posts/{id}/like.
2. Like Service inserts (post_id, user_id) into the Cassandra likes table; if it already exists it is a no-op (idempotent).
3. For a new like, INCR likes:{post_id} in Redis.
4. The like event goes to Kafka for notifications and analytics.
5. Every 5-10 seconds the Counter Aggregator persists the Redis counts in batches to the Cassandra counter table.
6. The client sees the updated count instantly through an optimistic UI.

C.7 Deep dives

Problem: the celebrity problem (an account with 10 crore followers). With fan-out on write one post = 10 crore Redis writes, taking minutes. So hybrid: push for normal users, pull at read time and merge for celebrities (for example >10 lakh followers).

Problem: fan-out on write vs fan-out on read. The push model has fast reads (precomputed) but expensive writes, wasted on inactive users. The pull model has cheap writes but queries N followees on every read. Hybrid is best: push for most, pull for celebrities.

Problem: the like counter of a hot post (lakhs of likes per minute). To avoid contention on one DB row use Redis INCR; for very hot keys shard the counter into N sub-keys and sum on read. Flush in batches to the durable store.

Problem: the timeline cache grows very large. Store only post IDs (8 bytes), not the whole post. Trim every feed at 500 entries, keep only active users' feeds in the cache; rebuild inactive ones from Cassandra at login.

Problem: globally unique, time-sortable post IDs. Snowflake-style 64-bit ID: timestamp + machine id + sequence. Unique without a central DB, and the ID itself gives time ordering, which is useful for cursor pagination.

Problem: a post still shows in the feed after an unfollow or post delete. Filter at read time: while hydrating, drop deleted posts and unfollowed authors. A background job cleans up the timelines; eventual consistency.

C.8 Trade-offs

C.9 Interview tips

• State the read:write ratio first: it is read-heavy, which makes the case for precomputing (fan-out on write).
• Raise the celebrity problem yourself and explain the hybrid model; this is what the interviewer wants to hear.
• Show why the timeline cache holds only IDs, and the memory math.
• Mention hot-key sharding and batch flush for the like counter.
• Do not forget to handle media on a separate path (pre-signed S3 + CDN).