Walkthrough 4 — Design a Chat App (WhatsApp)

# Design a Chat App (WhatsApp) ## The brief Build a real-time chat application supporting 1:1 messaging, group chats, and message delivery guarantees. ## Step 1: Requirements - 1:1 messages (text, media). - Group chats (up to 256 members). - Online/offline status. - Read receipts. - Sent/delivered/read indicators. - Real-time delivery (< 1 sec). - Offline message storage (delivered when user comes online). - 2B users; 100B messages/day. ## Step 2: Capacity estimation - 100B messages/day = 1.15M messages/sec average. Peak: 5M/sec. - Avg message size: 200 bytes → 20TB/day. - 1B concurrent connections (assume 50% of users online). ## Step 3: High-level architecture ``` [Mobile / Web Client] ↓ (WebSocket / persistent connection) [Connection Servers] ↓ (route messages) [Message Service] ↓ [Message DB] [Presence Service] [Group Service] ``` ## Step 4: Connection management Each online user has a persistent connection to a "Connection Server" (WebSocket or proprietary protocol): - Client opens WS to nearest gateway POP. - Connection Server tracks: user_id → connection_id mapping. - Heartbeat to detect disconnects. Why persistent connection (not polling): - Real-time delivery requires server-push. - Latency: < 100ms for message delivery. - Battery: less overhead than constant polling. Scaling: - 1B connections / 100K connections per server = 10K connection servers. - Sharded by user_id (consistent hash). - Connection Server is stateful per its users; ZooKeeper / etcd tracks routing. ## Step 5: Message flow (1:1) ``` 1. Alice's client sends: { to: "bob", text: "hi" } 2. Alice's Connection Server receives. 3. Save to Message DB (durable; ordered). 4. Look up Bob's connection (via routing service): - If Bob is online: push to his Connection Server → Bob's client. - If Bob is offline: store in "pending messages" for Bob. 5. Alice's client gets ACK: "delivered". 6. When Bob reads: client sends "read" event → updates DB → Alice's client notified. ``` ## Step 6: Message delivery guarantees - **At-least-once**: server retries until client ACKs. - Each message has a unique ID; client dedupes if received twice. - Order: messages from one sender to one recipient preserved (use ordered DB or per-conversation queue). ## Step 7: Storage Message DB: write-heavy, time-series-like. - Cassandra is a strong fit (partition by conversation_id, cluster by timestamp). - Each conversation = one partition. ``` conversation_id (partition key) → message_id, sender_id, content, timestamp, status (clustering columns) ``` For 100B messages/day × 200 bytes × 1 year = 7PB. Massive; sharded Cassandra cluster. Retention: keep all messages indefinitely (WhatsApp policy varies); cold storage for old. ## Step 8: Group chats Each group: - Group ID. - Member list (up to 256). - Permissions (admin, member). Sending to group: - Sender's connection server gets message. - Fan-out: write to group's message log; push to each online member. - Offline members: pending queue per user. For 256-member group: 256 deliveries per group message. Manageable. ## Step 9: Presence (online status) - Heartbeats from client every 30s. - Connection Server updates Redis: `user:123:last_seen = timestamp`. - "Online" if last_seen within 60s. Privacy: most apps allow hiding online status; presence check respects this. ## Step 10: Read receipts - When client displays a message: send "read" event. - Server updates message status. - Push update to sender (their UI shows "read"). For groups: read-by list (up to 256 entries per message; significant storage). ## Step 11: Media Photos/videos: - Upload to S3 (presigned URLs). - Message contains S3 URL + metadata. - Encrypted at rest (E2E if app supports). - Auto-delete from S3 after N days for offline-recovery; permanent only on user device. ## Step 12: End-to-end encryption (E2E) WhatsApp uses Signal Protocol: - Each pair of users has ephemeral session keys. - Server stores encrypted blobs only; can't decrypt. - Forward secrecy: keys rotate per message. System design implication: server can't index or search messages (privacy by design). Implementing E2E from scratch is hard; use Signal library. ## Step 13: Notifications (offline) If recipient is offline, push notification via APNS (iOS) / FCM (Android): - Notification contains sender name + preview (or encrypted token if E2E). - Tap → app opens → fetches actual message from server. ## Common follow-ups - **Voice/video calls?** Separate signaling service + WebRTC. - **Status / Stories?** Separate service; ephemeral content. - **Backup?** Encrypted backup to cloud (user's key). - **Multi-device?** Sync messages across devices (companion devices in WhatsApp). - **Spam / abuse?** Moderation pipeline; user reports; ML classifiers. ## Self-rubric - [ ] Clarified requirements (1:1, groups, online status, etc.) - [ ] Estimated scale (messages/sec, storage) - [ ] Designed WebSocket-based connection layer - [ ] Routing of online users v