Feed Architecture Overview
A social feed shows users content from people and pages they follow, ranked by predicted engagement. The pipeline has three stages: Candidate generation: retrieve the pool of candidate posts (from follows, interest graphs, trending topics). Ranking: score each candidate with an ML model predicting user engagement (click, like, share, comment, dwell time). Filtering and serving: apply business rules (diversity, safety, deduplication), truncate to top N, serve. At Instagram/TikTok scale: billions of posts, hundreds of millions of users, < 200ms feed load SLA. The key trade-off: coverage (seeing all relevant content) vs. latency vs. freshness.
Fan-Out Strategies: Push vs. Pull
Fan-out on write (push model): when a user posts, push the post ID to the feed inbox of all followers. Read: just read your pre-computed inbox. Write: O(followers) — expensive for celebrities (Cristiano Ronaldo has 600M followers). Fan-out on read (pull model): on feed load, fetch posts from all accounts the user follows, merge and rank. Read: O(accounts_followed * posts_per_account) — expensive for users following thousands. Hybrid (Instagram’s approach): For users with 10K followers): fan-out on read — inject celebrity posts at feed serving time. This caps write amplification while keeping reads fast for most users. Celebrity threshold is a tunable parameter.
Candidate Generation
class CandidateGenerator:
def get_candidates(self, user_id: int, limit: int = 500) -> list[Post]:
candidates = []
# 1. Follow graph: posts from accounts the user follows
follow_posts = self.follow_feed_store.get_recent(
user_id, limit=200
) # Redis sorted set: ZREVRANGE feed:{user_id} 0 199
# 2. Interest graph: posts from topics the user engages with
interest_posts = self.interest_store.get_recommended(
user_id, limit=150
) # Embedding similarity in vector DB
# 3. Trending: globally trending content
trending = self.trending_store.get_trending(
region=user.region, limit=50
)
# 4. Re-engagement: posts the user liked/shared that have new replies
reengagement = self.reengagement_store.get(user_id, limit=50)
# Deduplicate by post_id
seen = set()
for post in follow_posts + interest_posts + trending + reengagement:
if post.id not in seen:
candidates.append(post)
seen.add(post.id)
return candidates[:limit]Ranking Model
Features for the ranking model: Post features: content type (video/photo/text), post age, creator follower count, post engagement velocity (likes per hour), content embeddings. User features: historical engagement with this creator, content type preference, session context (time of day, device), recent interaction history. Cross features: user-creator affinity score, user-content-type affinity. Model: two-tower neural network (user tower + post tower → dot product for relevance score) or gradient-boosted trees for multi-task prediction (predict P(click), P(like), P(share), P(dwell > 10s) separately, combine into a single score). Multi-task loss prevents optimizing only for clicks (click-bait) — weight the combination based on business values.
Real-Time Feed Updates and Freshness
New posts should appear in feeds within seconds of publishing. Real-time injection: when a post is created, publish to a Kafka topic (new_posts). Feed service consumers process new_posts events and inject them into Redis sorted sets (fan-out on write). For pull-model users: the injection is skipped — the post is available for pull on next feed load. Freshness signals: recently posted content gets a “freshness boost” in the ranking score that decays over time (exponential decay: score * exp(-lambda * age_hours)). This ensures new posts from followed accounts surface even against older high-engagement posts. Seen-post filtering: track posts already shown to the user in a Bloom filter (per user, per session). Filter candidates against the Bloom filter before ranking to avoid showing the same post twice in consecutive feed loads.
See also: Meta Interview Prep
See also: Twitter/X Interview Prep
See also: LinkedIn Interview Prep