Instagram System Design Masterclass
Master the mechanics of high-speed social feeds. Explore how Instagram handles massive writes using **hybrid fan-out topologies (push vs. pull)**, compresses media assets via async transcoder workers, shards high-density followers lists across **PostgreSQL table partitions**, and compiled feeds within sharded **Redis timeline indexes**.
Ingestion vs. Feed Loading Planes
At massive scale, social media feeds have heavily asymmetrical traffic profiles. Reading the timeline is 100x more frequent than creating a post. Therefore, writing and reading operations must traverse completely isolated execution routes. This protects the timeline consumption route from database write locks during spike events, like a major live media broadcast or viral meme storm.
Core Challenges
- • Hybrid Fan-Out: Compiling timelines dynamically for regular users (push model) vs. celebrity accounts (pull model).
- • Graph Sharding: Scaling relational queries like `SELECT followers WHERE user_id = X` across sharded databases.
- • Media Compression: Transcoding uploads instantly into multi-bitrate H.264 video and WebP image formats.
Expected Scale Parameters
Hybrid Feed Fan-Out (Push-on-Write vs. Pull-on-Read)
When a normal user uploads a post, it is fanned out to their 500 followers by writing the post ID directly to each follower's pre-compiled **Redis timeline cache** (Push-on-Write). However, doing this for a celebrity account with 100M+ followers would trigger a massive write-amplification spike, blocking cache engines for minutes. Instagram resolves this using a **Hybrid Fan-Out Topology**.
Push-on-Write (Normal Users)
When a user with a low follower count publishes a post, the transaction engine triggers an async background worker. This worker fetches the user's followers list and inserts the post ID directly into each follower's compiled Redis timeline cache. When followers open the app, the timeline loads instantly in a single Redis query without heavy database joins.
Pull-on-Read (Celebrity Feeds)
For celebrity accounts with millions of followers, push fan-out is completely skipped to avoid write amplification. When a follower opens their feed, the app queries the follower's pre-compiled Redis timeline *and* dynamically queries active posts from celebrities they follow in real time. The **Merge Resolver** merges, sorts by timestamp, and serves the combined feed in under 15ms.
High-Speed Video Transcoding & CDN Ingestion
Serving media assets instantly to a global audience requires strict compression. Large, raw video files uploaded from modern phone cameras would saturate client bandwidth, triggering heavy buffer lags. Instagram resolves this by immediately routing uploads through an **Async Transcoder Pipeline** that outputs multi-bitrate H.264 formats, caching them on geo-distributed **Content Delivery Networks (CDNs)**.
How media asset distribution scales:
- Pre-Signed Upload URLs: Client applications request pre-signed upload URLs from the gateway to write directly to **Amazon S3 object stores**, keeping socket gateways isolated from heavy data streams.
- Event-Driven Transcoding: An S3 write event publishes to a Kafka topic, waking up dynamic **resizer and transcoder coroutines**. Images are optimized into WebP, and videos are sliced into multiple resolution chunks.
- Adaptive Bitrate Streaming (ABR): Sliced HLS chunks (ranging from 360p to 1080p) are cached on local CDNs. Player engines on phones dynamically request higher or lower resolution slices depending on active network speeds.
Redis Timeline Caching & Index Pools
Compiling a user's timeline by querying SQL indexes on every single scroll would create massive database bottlenecks. Instagram resolves this by keeping pre-compiled feeds cached inside sharded **Redis memory clusters** as sharded timeline log indexes.
How memory feed caching performs:
- • Active Cache Policies: Timelines are compiled *only* for active users who have opened the app within the last 14 days. Inactive users are pruned from memory automatically, saving massive hardware overheads.
- • Redis Sorted Sets (ZSET): Feeds are stored in Redis as Sorted Sets (`ZSET`), using the **Post Timestamp** as the index score. This allows fetching the exact next 20 items (`ZREVRANGEBYSCORE`) in under 1.5ms.
Dynamic Activity Notification Ingestion (Kafka Streams)
When a popular creator posts, we must notify all their active followers in real time. Processing million-recipient notifications synchronously on the main thread would crash gateway nodes instantly. Instagram resolves this by passing activities to **Apache Kafka queues**, utilizing partition-keyed streams to distribute tasks safely across fan-out workers.
Real-time Fan-out Dispatcher
When a new post event is processed by the fan-out router, it is immediately written to **Apache Kafka partition queues**. Hashed by Owner UUID, these queues guarantee partition isolation. Decoupled **notification worker coroutines** read events in strict sequence, query user followers caches from Redis, and dispatch push packets via Apple APNS or Google FCM in under **25ms**.
Architect Interview Q&A
Q1: How does Instagram prevent massive write amplification when a celebrity posts?
If we fanned out a post by a celebrity with 100M+ followers using push-on-write, we would have to execute 100 million cache updates immediately. This would crash Redis. Instagram avoids this using a **Hybrid Fan-Out Model**. Celebrity posts are stored *only* in the celebrity's local post cache. When followers open the app, the system queries the celebrity's cache dynamically (pull-on-read) and merges it with the follower's pre-compiled feed in under 15ms.
Q2: Why shard PostgreSQL tables by Owner User ID instead of Post UUID?
Sharding by Post UUID would distribute a user's comments and likes randomly across multiple database instances. Loading a profile would require querying every single database instance in the cluster, creating huge network lags. Sharding by **Owner User ID** guarantees that all relational graph items (likes, comments, followers) for a user live within the exact same database shard, keeping lookup latency to sub-2ms.
Q3: How are pre-compiled timeline caches stored inside Redis efficiently?
Pre-compiled timelines are cached as **Redis Sorted Sets (ZSET)**, using the **Post Timestamp** as the sorting score. This enables sub-1.5ms timeline pagination (`ZREVRANGEBYSCORE`). To keep memory footprint low, caches are built *only* for active users who have opened the app within the last 14 days; inactive users are automatically pruned from memory.
Q4: How does Adaptive Bitrate Streaming (ABR) reduce media loading times on slow networks?
When a video is uploaded, our transcoder slices it into small 2-second segments across multiple bitrates (360p, 480p, 720p, 1080p). The client app's media player dynamically monitors active bandwidth and requests higher or lower resolution slices on the fly, preventing playback pauses even under unstable network conditions.
Unified Instagram Activity Blueprint
Here is the end-to-end media upload and timeline assembly blueprint, detailing the journey of an uploaded post from the sender's device, through S3 buckets, transcoder resizers, hybrid fan-out pipelines, sharded Redis timeline stores, and global CDNs, to the receiver's screen.
Unified Feed Assembly Walkthrough:
Media Upload Initiated
Alice gets a pre-signed S3 URL from the gateway and uploads her photo/video directly to S3 object storage.
Async Transcoder Wakened
The upload triggers a transcoder worker coroutine, compressing images into WebP and slicing videos into multiple resolution chunks.
CDN Distribution Edge Caching
Optimized media chunks publish directly to geo-distributed CDN caching edges, keeping loading latency sub-30ms global.
Dynamic Fan-out Evaluation
The fan-out engine evaluates Alice's follower metrics, dynamically choosing push model for normal users vs. pull model for celebrities.
Redis timeline updates
For normal accounts, the background worker writes the post ID directly to followers' Redis timeline cache indices.
Sharded PostgreSQL Graph writes
Simultaneously, the main post metadata writes to owner-keyed PostgreSQL shards, ensuring relational safety.
Dynamic Feed Merging
When Bob opens his feed, the app queries his pre-compiled timeline, dynamically merges celebrity feeds, and renders the combined feed in under 15ms.