Design WhatsApp

Design a messaging app like WhatsApp. Users can send text, images, and videos to individuals or groups. Messages must be delivered in order and receipts (sent/delivered/read) must be tracked.

Capacity Estimation

• Message rate: 100B messages/day = 1.16M messages/sec
• Message size: avg 100 bytes text. 100B × 100 bytes = 10 TB/day text storage
• Media: ~20% messages have media. Stored on S3; metadata only in DB
• Active connections: 500M concurrent WebSocket connections

Architecture

[Client A] ──── WebSocket ────▶ [Chat Server]
                                      │
                              [Message Queue (Kafka)]
                                      │
                            ┌─────────┴──────────┐
                     [Delivery Worker]    [Push Notification Service]
                            │                         │
                    [Client B online?]         [APNs / FCM]
                         Yes │
                    [Client B WebSocket]

Each chat server maintains WebSocket connections. A user might connect to any server — a routing layer (via consistent hashing on userId) directs messages to the right server.

API Design

WebSocket messages (not HTTP):

// Send message
{ "type": "MESSAGE", "to": "userId", "content": "Hello", "clientMsgId": "uuid" }

// Delivery receipt (from server)
{ "type": "DELIVERED", "msgId": "...", "to": "userId" }

// Read receipt
{ "type": "READ", "msgId": "...", "by": "userId" }

REST API for non-real-time:

GET /messages/{conversationId}?before=cursor&limit=50
POST /media/upload → presigned S3 URL
GET /users/{userId}/presence

Database Design

Messages (Cassandra — write-heavy, time-series)

messages
  conversation_id UUID    (PK partition key)
  message_id      TIMEUUID (PK clustering key, newest first)
  sender_id       UUID
  content         TEXT
  media_url       TEXT    nullable
  status          ENUM    (SENT, DELIVERED, READ)
  created_at      TIMESTAMP

Why TIMEUUID for clustering? Guarantees ordering by time AND uniqueness — no two messages have the same ID even if created at the same millisecond.

Conversation metadata (PostgreSQL)

conversations: id, type (1:1 | GROUP), created_at
participants: conversation_id, user_id, joined_at

Presence (Redis, TTL-based)

SETEX presence:userId 30 "ONLINE"  # refreshed every 20s by heartbeat

Scaling

WebSocket connection scaling

500M concurrent WebSockets cannot live on one server. Each chat server handles ~50K connections. We need ~10,000 chat servers.

A routing service (Redis pub/sub or a service mesh) tracks which server each user is connected to:

REDIS: user:{userId}:server → "chatserver-047"

When Server A needs to deliver to a user on Server B, it publishes to Redis pub/sub channel for that user, and Server B picks it up.

Message ordering

Use Cassandra TIMEUUID + per-conversation sequence numbers. The client reorders on display.

• WebSocket vs long-polling: WebSocket is bidirectional and low-latency. Long-polling is simpler but wasteful.
• Cassandra vs PostgreSQL for messages: Cassandra scales writes horizontally; PostgreSQL is easier but harder to shard for 100B msgs/day.
• Fan-out vs pull for group messages: Fan-out to each member (1000 copies) is expensive for large groups. For 1000-member groups, fetch-on-read is more practical.

• Hot conversations — rate-limit message sending per conversation
• Media upload — use presigned S3 URLs; don't proxy media through chat servers
• Notification fan-out — for large groups, batch and de-duplicate push notifications

• Chat server crash: client reconnects via WebSocket to any server. Messages queued in Kafka until delivery.
• Kafka lag: delivery delayed but not lost — Kafka retains messages for 7 days.
• DB unavailable: writes buffered in Kafka, processed when DB recovers.