Log in
SkillPilotAI
Plans

Case Study 3: WhatsApp Chat & Messaging

Real-time 1:1 and group chat among hundreds of millions of people, with tick marks, online status, media and offline delivery.

C.1 Requirements

Functional requirements:

  • 1:1 chat: send and receive text messages instantly.
  • Group chat: fan out a message in groups of up to 1,000 members.
  • Receipts: single tick (reached the server), double tick (delivered to the device), blue tick (read).
  • Online status and last seen; a "typing..." indicator.
  • Media: send photos, videos, documents, voice notes (upload once, downloaded by receivers).
  • Offline messages: if the user is offline the message stays stored and is delivered as soon as they come online.
  • End-to-end encryption: the server cannot read message content (WhatsApp publicly uses the Signal protocol).

Non-functional requirements:

  • Low latency: message delivery between online users p99 < 200-300 ms.
  • High availability 99.99%; when a chat server goes down, users reconnect to another server.
  • Ordering: messages within a chat must appear in the right order; no message may be lost or duplicated.
  • Scale: 50 crore (500 million) DAU, ~20 crore (200 million) concurrent connections, lakhs of messages/second.
  • Mobile friendly: low battery and data use, works even on a weak 2G/3G network.

C.2 Estimation (back-of-the-envelope)

ItemNumberHow it is derived
Daily active users50 croreDesign assumption; WhatsApp publicly reports 2 billion+ users.
Messages per day2,000 crore (20 billion)50 crore users x 40 messages/day = 2,000 crore.
Message QPS~2.3 lakh/sec avg, ~7 lakh peak2,000 crore / 86,400 = ~2.3 lakh/sec; peak 3x (festival wishes, New Year).
Concurrent connections~20 crore40% of DAU online at once; if one chat server handles ~10 lakh connections that is ~200 servers, ~500 with headroom.
Text storage (transient)~2 TB/dayAverage message 100 bytes x 2,000 crore = 2 TB/day; deleted from the server after delivery, it stays only in the offline queue.
Media storage~400 TB/day10% of messages are media = 200 crore x ~200 KB avg (compressed) = 400 TB/day in object storage.
Text bandwidth~2 Gbps avg2.3 lakh msg/sec x ~1 KB (encryption + headers) = 230 MB/s = ~1.8 Gbps; media comes on top of this via the CDN.

C.3 API design

EndpointPurpose
WSS /v1/connectPersistent WebSocket/TCP connection with an auth token; all messages travel over it
SEND {msg_id, to, chat_id, ciphertext, ts}Socket frame: send a message; dedup using the client-generated msg_id
ACK {msg_id, type: delivered|read}Socket frame: receipt back to the sender
POST /v1/mediaEncrypted media upload; the response carries media_id/URL (S3 + CDN)
GET /v1/media/{media_id}The receiver downloads the encrypted media (from the CDN)
POST /v1/groups/{id}/membersAdd/remove a group member (admin only)

C.4 Data model

Table / StoreFieldsWhy
messages / offline queue (Cassandra/HBase, partition = user_id)recipient_id PK, msg_id CK (time-ordered), sender_id, chat_id, ciphertext, media_ref, tsWrite-heavy; a user's pending messages sit in one partition; deleted after delivery.
sessions (Redis)user_id -> chat_server_id, connection_id, last_heartbeatTo route a message we must know which server the user is connected to.
presence (Redis with TTL)user_id -> online/last_seen_tsVery frequent updates; goes offline automatically through the TTL.
groups (MySQL/Cassandra)group_id PK, name, created_by; group_members: group_id, user_id, roleMembership list used for fan-out; fast read by group_id, also cached.
users (MySQL, sharded by user_id)user_id PK, phone UNIQUE, name, public_keys, privacy_settingsRegistration by phone number, distributing encryption public keys.
media (S3/blob storage)media_id key, encrypted blob, size, expiryLarge binary files in cheap object storage, served through the CDN.

C.5 High-level architecture

WhatsApp: architecture and request flow
  1. 1.Clients: User A mobile app (sender), WhatsApp Web / Desktop, User B mobile app (receiver)
  2. 2.DNS (geo, nearest DC) -> L4 Load Balancer (long-lived TCP / WebSocket, sticky by connection)
  3. 3.Chat Server 1 (A's WebSocket, lakhs of connections) and Chat Server 2 (B's WebSocket, Erlang); a separate API Gateway (HTTP) for media and groups
  4. 4.Services: Message Router (which server is B on?), Presence (heartbeat), Group Service (members), Media Service (encrypted upload)
  5. 5.Data: Cassandra (offline msgs, recipient_id + msg_id), Redis (sessions user -> chat server, presence TTL), MySQL (users, public keys, groups), S3 + CDN (media blobs)
  6. 6.B offline / group: Kafka (offline push events, group fan-out events)
  7. 7.Workers: Push Notification Worker (FCM / APNs wakes B's phone), Group Fan-out Worker (online push / offline queue per member), Media Cleanup Job (expiry)

Ticks: single = the server saved it durably, double = delivered, blue = READ ACK. Scale: 20 crore concurrent connections, ~500 chat servers, 2,000 crore msgs/day, ~7 lakh/sec peak.

ComponentWhat it does
Chat ServerHolds lakhs of persistent WebSocket connections; WhatsApp is publicly known for its Erlang/FreeBSD stack.
L4 Load BalancerDistributes long-lived TCP connections across chat servers; sticky by connection.
Session Registry (Redis)user_id -> which chat server; the phone book for message routing.
Message RouterTakes the message from the sender's server to the recipient's server, or puts it in the offline queue.
Offline Store (Cassandra)Keeps undelivered messages durable until the recipient ACKs.
Group ServiceProvides the group member list; the fan-out worker sends a copy to every member.
Presence ServiceUpdates online/last seen via heartbeat; only contacts who have the chat open are subscribed and updated.
Media ServiceThe client uploads encrypted media to S3; only the media URL + key travel in the message.
Push Notification WorkerSends an FCM (Android)/APNs (iOS) notification to an offline user so the app wakes up and syncs.

C.6 Main flows

1:1 message flow (with ticks):

  1. A's app encrypts the message with recipient B's public key and SENDs it over the WebSocket with a unique msg_id.
  2. A's chat server saves the message in Cassandra (or a durable queue) and sends A a server ACK: single tick.
  3. The Message Router looks up B's session in Redis: which chat server B is connected to.
  4. If B is online the message is forwarded to B's chat server, which pushes it on B's socket.
  5. B's app decrypts the message and sends a DELIVERED ACK; the server forwards it to A: double tick.
  6. After delivery the server deletes the message from the offline store.
  7. When B opens the chat a READ ACK is sent; A sees the blue tick (if read receipts are on).
  8. If a network retry causes a duplicate, it is deduplicated by msg_id and the user sees a single message.

Delivering a message to an offline user:

  1. The router finds no session for B in Redis, so B is offline.
  2. The message stays stored in Cassandra in B's partition with (recipient_id, msg_id).
  3. The push worker sends a notification to B's phone via FCM/APNs (without content, or encrypted).
  4. B's app wakes up, connects to a chat server and a new session is registered in Redis.
  5. The server pushes all of B's pending messages in order (sorted by msg_id).
  6. B sends an ACK for every batch; the server sends delivered receipts to A and deletes the pending messages.

Group message flow:

  1. A sends a message to group G; the client uses efficient encryption for every member (Signal "sender keys").
  2. The chat server stores the message once and puts a group fan-out event on Kafka.
  3. The fan-out worker fetches G's member list from the Group Service/cache (max ~1,000).
  4. Session lookup for every member: if online, push to their chat server; if offline, offline queue + push notification.
  5. Each member's delivered/read ACK comes separately; the sender sees per-member status in "message info".
  6. The double tick shows only once the message is delivered to all members.

C.7 Deep dives

1:1 message and ticks (A -> B, end-to-end encrypted)
  1. 1.A app -> Chat Srv A: SEND msg_id
  2. 2.Chat Srv A -> Cassandra: durable save; server ACK to A (single tick)
  3. 3.Chat Srv A -> Router + Redis: where is B's session?
  4. 4.Router -> Chat Srv B: forward; Chat Srv B -> B app: socket push
  5. 5.B app: decrypt, DELIVERED ACK -> A (double tick); server deletes the offline copy
  6. 6.B opens the chat: READ ACK -> A (blue tick)

If a retry brings a duplicate it is deduped by msg_id: the user sees exactly-once while the network is at-least-once.

If B is offline
  1. 1.No session: B is not in Redis = offline
  2. 2.Cassandra: in B's partition with recipient_id, msg_id
  3. 3.Push worker: FCM / APNs notification without content
  4. 4.B app wakes: reconnects + new session in Redis
  5. 5.Pending push: all messages in msg_id sorted order
  6. 6.Batch ACK: delivered receipt to A + delete from offline
Group message fan-out
  1. 1.A -> group G, sender keys; the chat server stores the message once
  2. 2.Kafka: group fan-out event -> Fan-out Worker
  3. 3.Members list from Redis cache (max ~1,000)
  4. 4.Online member: push to their chat server; offline member: offline queue + push notification
  5. 5.Double tick only after delivery to all members

Problem: handling 20 crore persistent connections. Every chat server holds lakhs of WebSocket connections (epoll/Erlang lightweight processes). Use an L4 LB, clean up dead connections with heartbeats, and after a server crash the client reconnects with exponential backoff and registers a new session.

Problem: ordering and showing exactly-once. The network delivers at-least-once; we dedup with the client-generated msg_id so the user sees exactly-once. Order within a chat is maintained with a per-chat sequence number or a time-ordered ID.

Problem: not losing a message (server crash). The single tick appears only after the message is written to a durable store (replicated Cassandra). It is not deleted until the recipient ACKs, so redelivery is possible even after a crash.

Problem: group fan-out load. One message to 1,000 members = 1,000 deliveries. So fan-out is done by async Kafka workers, media is uploaded once and only the reference is fanned out. The group member list is cached in Redis.

Problem: presence/online status storm. Sending every status change to all contacts is very costly. Updates go only to those who currently have that user's chat open (subscribe model); last seen is read lazily, and Redis TTL takes the user offline automatically.

Problem: server design with end-to-end encryption. The server only routes ciphertext; public keys are distributed through the server (Signal protocol). So server-side search or content moderation is not possible; this is a design constraint.

C.8 Trade-offs

DecisionWhat was chosen and why
WebSocket vs HTTP long pollingPersistent WebSocket/TCP: real-time, low overhead. Polling is simple but costs more battery and server load.
Server pe messages permanently store karein ya nahiDelete after delivery (WhatsApp publicly says this); better for both storage cost and privacy, with backups in the user's cloud.
Cassandra vs MySQL for messagesCassandra: heavy writes, partition by user, easy horizontal scaling. MySQL for relational data (users, groups).
Group fan-out on write vs on readFan-out on write for small groups (max ~1,000), so delivery is fast. For very large broadcast channels, fan-out on read is better.
Push vs pull for presencePush only for the active chat, the rest pull/lazy. Full push to every contact would generate far too much traffic.

C.9 Interview tips

  • Explain the connection model first: persistent WebSocket, session registry and how message routing works.
  • Explain the ticks (sent, delivered, read) through the ACK flow; interviewers like this a lot.
  • Be sure to cover offline delivery and the push notification (FCM/APNs) flow.
  • Keep the difference between ordering, dedup (msg_id) and at-least-once delivery clear.
  • Mention the tricks to scale group fan-out and presence; also mention the design impact of E2E encryption.