Walkthrough 4 — Design a Chat App (WhatsApp)

The brief

Build a real-time chat application supporting 1:1 messaging, group chats, and message delivery guarantees.

Step 1: Requirements

1:1 messages (text, media).
Group chats (up to 256 members).
Online/offline status.
Read receipts.
Sent/delivered/read indicators.
Real-time delivery (< 1 sec).
Offline message storage (delivered when user comes online).
2B users; 100B messages/day.

Step 2: Capacity estimation

100B messages/day = 1.15M messages/sec average. Peak: 5M/sec.
Avg message size: 200 bytes → 20TB/day.
1B concurrent connections (assume 50% of users online).

Step 3: High-level architecture

[Mobile / Web Client]
   ↓ (WebSocket / persistent connection)
[Connection Servers]
   ↓ (route messages)
[Message Service]
   ↓
[Message DB]
[Presence Service] [Group Service]

Step 4: Connection management

Each online user has a persistent connection to a "Connection Server" (WebSocket or proprietary protocol):

Client opens WS to nearest gateway POP.
Connection Server tracks: user_id → connection_id mapping.
Heartbeat to detect disconnects.

Why persistent connection (not polling):

Real-time delivery requires server-push.
Latency: < 100ms for message delivery.
Battery: less overhead than constant polling.

Scaling:

1B connections / 100K connections per server = 10K connection servers.
Sharded by user_id (consistent hash).
Connection Server is stateful per its users; ZooKeeper / etcd tracks routing.

Step 5: Message flow (1:1)

1. Alice's client sends: { to: "bob", text: "hi" }
2. Alice's Connection Server receives.
3. Save to Message DB (durable; ordered).
4. Look up Bob's connection (via routing service):
   - If Bob is online: push to his Connection Server → Bob's client.
   - If Bob is offline: store in "pending messages" for Bob.
5. Alice's client gets ACK: "delivered".
6. When Bob reads: client sends "read" event → updates DB → Alice's client notified.

Step 6: Message delivery guarantees

At-least-once: server retries until client ACKs.
Each message has a unique ID; client dedupes if received twice.
Order: messages from one sender to one recipient preserved (use ordered DB or per-conversation queue).

Step 7: Storage

Message DB: write-heavy, time-series-like.

Cassandra is a strong fit (partition by conversation_id, cluster by timestamp).
Each conversation = one partition.

conversation_id (partition key)
  → message_id, sender_id, content, timestamp, status (clustering columns)

For 100B messages/day × 200 bytes × 1 year = 7PB. Massive; sharded Cassandra cluster.

Retention: keep all messages indefinitely (WhatsApp policy varies); cold storage for old.

Step 8: Group chats

Each group:

Group ID.
Member list (up to 256).
Permissions (admin, member).

Sending to group:

Sender's connection server gets message.
Fan-out: write to group's message log; push to each online member.
Offline members: pending queue per user.

For 256-member group: 256 deliveries per group message. Manageable.

Step 9: Presence (online status)

Heartbeats from client every 30s.
Connection Server updates Redis: user:123:last_seen = timestamp.
"Online" if last_seen within 60s.

Privacy: most apps allow hiding online status; presence check respects this.

Step 10: Read receipts

When client displays a message: send "read" event.
Server updates message status.
Push update to sender (their UI shows "read").

For groups: read-by list (up to 256 entries per message; significant storage).

Step 11: Media

Photos/videos:

Upload to S3 (presigned URLs).
Message contains S3 URL + metadata.
Encrypted at rest (E2E if app supports).
Auto-delete from S3 after N days for offline-recovery; permanent only on user device.

Step 12: End-to-end encryption (E2E)

WhatsApp uses Signal Protocol:

Each pair of users has ephemeral session keys.
Server stores encrypted blobs only; can't decrypt.
Forward secrecy: keys rotate per message.

System design implication: server can't index or search messages (privacy by design).

Implementing E2E from scratch is hard; use Signal library.

Step 13: Notifications (offline)

If recipient is offline, push notification via APNS (iOS) / FCM (Android):

Notification contains sender name + preview (or encrypted token if E2E).
Tap → app opens → fetches actual message from server.

Common follow-ups

Voice/video calls? Separate signaling service + WebRTC.
Status / Stories? Separate service; ephemeral content.
Backup? Encrypted backup to cloud (user's key).
Multi-device? Sync messages across devices (companion devices in WhatsApp).
Spam / abuse? Moderation pipeline; user reports; ML classifiers.

Self-rubric

Clarified requirements (1:1, groups, online status, etc.)
Estimated scale (messages/sec, storage)
Designed WebSocket-based connection layer
Routing of online users via connection server map
Offline message handling
Read receipts and delivery indicators
Group chat fan-out
Mentioned E2E encryption considerations
Storage strategy (Cassandra for messages)

Deliverables

Architecture diagram (connection layer, message service, storage).
Message flow diagrams (online → online; online → offline).
Data model for messages and groups.
Scaling discussion (connection server fleet, message DB sharding).

Assignment Instructions