eBay System Design Masterclass
Deconstruct the systems that power millisecond auction updates. Explore how eBay achieves strict bidding sequence serialization using **partition-keyed Kafka topics**, handles concurrent auction locks using **Redis Redlock consensus**, and broadcasts active bids via high-throughput WebSockets.
The Live Bidding Scale
Live e-commerce bidding presents a complex concurrent transactions challenge. While catalog navigation is highly read-intensive, the closing seconds of a popular auction trigger hundreds of thousands of concurrent writes on a **single item record**. Standard relational databases immediately choke under lock queues. Building this at scale requires strict queue sequencing, in-memory distributed locks, and real-time WebSocket broadcasting meshes.
Core Challenges
- • FIFO Sequencing: Guaranteeing that bids on an item are processed in absolute chronological order down to the millisecond.
- • Atomic Bid Settle: Acquiring dynamic locks on individual items to prevent double-bids or stale-bids.
- • Real-Time Broadcast: Fanning out the winning bid value instantly to all active watchers under 50ms.
- • Proxy Bidding Engine: Managing automated bids matching against player max caps without database locks.
Expected Scale Parameters
Concurrent Bidding Locks (Redis Redlock Consensus)
When thousands of users submit a bid on the same item during its final 3 seconds, writing directly to a database creates massive locking queues. eBay prevents this by placing an **in-memory distributed lock** on the active item ID SKU inside sharded **Redis clusters**, utilizing the **Redlock Consensus Algorithm** to ensure atomic reservation.
Consensus Lock Routing
When a bid is processed, the coordinator attempts to write a short-lived key containing the bid token (`SET item_uuid bid_token NX PX 500`) across 3 sharded Redis master instances in parallel. If at least 2 out of the 3 instances successfully acknowledge the write, consensus is met, the lock is acquired, and the transaction is committed, guaranteeing absolute safety from concurrent bid-race conditions.
Dynamic TTL Management
The lock has an extremely short TTL (Time-To-Live) of 500ms. If a payment check fails or network drops occur, the lock expires automatically, returning the active item to availability pool instantly.
FIFO Bid Sequencing & Partition Queues
Deciding the absolute winner in the final millisecond requires strict first-in-first-out (FIFO) sequencing. Bids cannot be processed in random multithreaded pathways; they must traverse highly ordered partition channels inside **Apache Kafka topics**.
How ordered sequencing operates:
- Hash-Key Partitioning: Every incoming bid request is routed to Kafka using the unique **Item UUID** as the partition hash key. This guarantees that all bids on a specific item end up in the exact same Kafka partition.
- Sequential Log Consumption: Within a partition, Kafka guarantees absolute ordering. Go-based partition consumers process bids one-by-one sequentially, preventing out-of-order race conditions.
- Validation Engine: The consumer thread checks: "Is this bid higher than the active bid + increment value?" If yes, it is committed; if no, it is discarded instantly.
Millisecond Bids Broadcast (WebSocket Mesh)
Once a bid is committed to the database, the new price must be broadcast to all active watchers on that item instantly to trigger the next round of bids. Standard HTTP polling introduces 5-second lags; eBay resolves this using a **WebSocket Broadcaster Fan-Out Mesh**.
High-Throughput Broadcaster Fan-out
When the state engine commits a new bid, it dispatches a lightweight JSON event payload to the **Kafka Broadcast Topic**. Decentralized **Go WebSocket gateway nodes** consume the event, locate the active websocket connections subscribed to that item UUID, and push the price update down open channels to all active screens in under **35ms**, resulting in continuous, responsive updates.
High-Speed Bid Settlement Ledger
Committing bid states to persistent storage requires absolute consistency while avoiding write latency. eBay utilizes sharded **PostgreSQL database databases** indexed by Item UUIDs combined with dynamic write buffers to achieve millisecond transaction execution.
Sharded Ledger Settlements
When the state engine validates a bid, the transactional details (bid UUID, bidder account, dynamic values, timestamp) are written directly to **sharded PostgreSQL database databases** partitioned by item UUIDs.
To prevent database read-locks from choking active writes, concurrent read queries (such as item auctions loading pages) are routed directly to high-capacity **Redis caches** containing active bid indices. When a transaction commits, the Redis cache is updated instantly in under 1.2ms, allowing database disk transactions to execute sequentially in the background.
Automated Proxy Bidding Engine
A massive portion of live bidding traffic is generated by automated proxy bidding bots (which immediately raise the price when other users submit a bid up to a predefined limit). Executing these automated loops over standard DB writes creates infinite circular loops. eBay solves this using an **In-Memory Proxy Bidding Engine**.
Decoupled Proxy Calculations
When you set a maximum bid of $500 on a $200 item, the system does not place a $500 bid instantly.
Instead, your max cap is encrypted and cached inside highly available **Redis clusters**.
When another user places an active bid of $210, the **Proxy Bidding Engine** intercepts the request, reads your Redis bot cap, computes the required delta increment (e.g. $215), and commits the auto-raised bid instantly in under **< 2.5ms**.
This entirely in-memory evaluation avoids database roundtrips, allowing thousands of bot updates to settle under peak seconds.
Expert Interview Q&A
Prepare for elite system design loops. Study the exact technical distributed locking and bid sequencing questions and production-ready answers.
Q1. How does the system handle an instantaneous connection crash during an active lock reservation?
Answer: By applying short-lived lease durations to Redis distributed locks (**500ms TTL**). If the network crashes mid-transaction, the lock expires automatically, returning the active item to availability pool without blocking other bids.
Q2. Why is partition-keyed sharding selected in Kafka over standard concurrent queues?
Answer: Partition-keyed sharding using Item UUIDs ensures that all bids on a specific item always land in the exact same Kafka partition. Within a partition, Kafka guarantees absolute chronological ordering, allowing Go partition consumers to process bids sequentially and prevent concurrent out-of-order race conditions.
Q3. How is database thread starvation prevented during peak closing seconds?
Answer: By separating write execution pipelines from active read traffic. Bids settle asynchronously on sharded PostgreSQL databases. Active read requests (e.g. watchers loading auction pages) are routed directly to high-capacity **Redis caches** containing active bid indices. When a transaction commits, the cache is updated instantly in under 1.2ms, allowing datastores to execute writes in the background.
Q4. How does the system scale proxy bots calculations without database lockups?
Answer: By computing bot updates **in-memory**. User maximum caps are cached inside Redis clusters. When other users place a bid, the in-memory **Proxy Bidding Engine** reads the cap, computes the delta increment, and commits the auto-raised bid in under 2.5ms without executing database lockups.
Unified Bidding Blueprint
Below is the ultimate, unified architecture topology powering eBay's real-time live bidding network. It traces bidder clicks to partitioned Kafka queues, in-memory distributed locks, proxy bidding engines, and high-capacity WebSocket broadcasters.
Live Bidding Lifecycle: Step-by-Step
User Bid Submission (Ingress Gateway Ingest)
When a user submits a bid, a secure session opens via regional Ingress gateways. Gateways decrypt SSL, validate user sessions, and check rate-limiting tokens in under 3ms.
Distributed Lock Acquisition (Redis Redlock Consensus)
The coordinator attempts to write the bid key across 3 sharded Redis master instances in parallel. If at least 2 instances acknowledge the write, consensus is met, the lock is acquired, and the transaction proceeds.
Kafka FIFO Sequencing
Approved bids publish to highly ordered **Kafka partitions** sharded by Item UUIDs. Partition consumers process bids sequentially, preventing out-of-order race conditions.
Bid Valuation & State Commitment
The state engine consumer thread validates if the bid is higher than the active bid + increment value. If valid, the new active bid state is committed.
In-Memory Proxy Bidding Evaluation
In parallel, the **Proxy Bidding Engine** reads encrypted maximum caps from Redis, computes required delta increments, and commits the auto-raised bid in under 2.5ms without executing database locks.
Sharded DB Ledger Settlement
Approved transactional details write directly to **sharded PostgreSQL database shards** partitioned by Item UUIDs in the background, keeping disk transactions decoupled.
WebSocket Broadcast Fan-out
The committed bid price publishes to a **Kafka Broadcast Topic**. Go-based WebSocket gateways consume the event and push the price update down open channels to all active watchers in under 35ms.