Chat app: order, offline users, big groups and what breaks

Part of the free System Design course on LogicWiz, in the chapter: Design a chat app (WhatsApp). The interactive whiteboard exercises and simulations run in the app.

Week 12: Replies above the questions

"In the group chat, everyone saw Ravi's 'Yes!' above Asha's 'Shall we leave at six?'. Three phones, three clocks, three opinions."

A tester's bug report on the first build

Deep dive 1: keeping order inside a conversation

Users never ask for one global order of all messages in the world. They need the same order inside one conversation, for every member. Say that scope first. It makes the problem much smaller.

Weak: order by the sender's phone clock. Phone clocks differ by seconds or minutes, so replies appear above questions. Server timestamps are better but still unsafe: two message-service instances can stamp the same millisecond, and their clocks drift too.
Good: a sequence number per conversation, given by the server. Messages in one conversation get seq 4,811, 4,812, 4,813. Clients sort by seq. A gap is easy to see: a phone that holds 4,810 and receives 4,812 knows to ask for 4,811. The counter is an atomic increment (it cannot be split, so two writers never get the same number).
Great: a single writer per conversation. Route all sends of a conversation to the same owner process, chosen by hashing the conversation id. That process assigns seq, saves the message, and publishes it to the stream partition for that conversation. No two processes fight over a lock, and order holds all the way to delivery. If the owner dies, another takes over from the stored last seq, and a conditional write on (conversation, seq) rejects a stale owner.

The trade-off: you give up a global order, which nobody needs. One very busy conversation is limited to the speed of one partition, which is plenty for 100 people. The seq also serves as the sync bookmark and as the read pointer. Retries never get a second seq, because a repeated client_msg_id is dropped, and one writer means two retries cannot race.

Deep dive 2: offline users and the backlog

Ravi's phone is off all night. Asha and her friends keep writing. What should happen?

Weak: deliver only over the live socket. No socket means no delivery. Messages held only in a gateway's memory also vanish when that server restarts.
Good: store first, then try to deliver, then push a notification. The message is in the store before the tick. The worker finds no session entry for Ravi, sends a push notification, and leaves the message in storage.
Great: make the pull path the guarantee. Every phone remembers the last seq it holds for each conversation. On reconnect it calls sync with those numbers and pulls everything after them. Live pushes become a speed-up, not the promise.

Meera said, "So the push notification delivers the message." It does not. A push notification is only a nudge. It can be late or dropped, and nobody can promise otherwise. The message comes from sync. A lost push, a crashed worker or a restarted gateway can delay a message but never lose it.

Size the backlog, the messages waiting for a returning phone. Ravi is in 20 active conversations and each got 100 messages overnight. That is 20 x 100 = 2,000 messages, times 200 bytes, about 400 KB. Small. Send the newest page of each conversation first, so the inbox appears at once, and fetch the rest in pages of 50.

Delivery is at-least-once: a message may arrive by push and by sync. The phone removes duplicates by message_id. The trade-off is the burst of reads when many phones return together. A gateway deploy drops its 50,000 phones. If all return in the same second, that is 50,000 syncs at once. With a random delay (jitter) spread over 30 seconds, it is about 1,700 a second.

Now write the delivery promise in your own words.

Deep dive 3: big groups and presence

Fan-out means copying one message to every recipient. In a group of 100, one message is 100 deliveries. Say 1 message in 100 goes to a full group at the 80,000 a second peak. That is 800 x 100 = 80,000 deliveries a second from groups alone.

Weak: write a copy of the message into every member's mailbox. Storage and writes grow 100 times, and an edit or delete must touch 100 copies.
Good: store the message once and fan out only the delivery. A worker looks up all members in the session store and groups them by gateway. If the 100 members sit on 40 gateways, that is 40 calls, not 100. Receipts stay one number per member.
Great: treat size as a switch. At 100 members, pushing is fine. At 10,000 members, a chat with 5 messages a second asks for 50,000 deliveries a second from one conversation. For such groups, push only a light "new messages up to seq N" signal to people with the chat open. Everyone else fetches when they open it. The cost is a small delay for people not looking.

Presence means showing who is online and when they were last seen. Meera said it is one bit per user, so it is cheap. Do the sum. 10 million online users sending a heartbeat every 30 seconds is about 333,000 renewals a second. A sharded in-memory store can handle that across several nodes, and the message database could not. So presence lives in the session store. Write "last seen" to durable storage only when a connection closes, not on each heartbeat. And never push every change to all contacts: a user with 500 contacts going online would cause 500 pushes. Let the app ask for presence only for people on the screen.

What breaks, and how you notice

A gateway crashes. Its 50,000 phones reconnect elsewhere with jitter and sync. You see open connections drop and reconnects spike.
The stream falls behind. Messages are safe in storage but arrive late. Watch consumer lag and send-to-deliver p99, then add workers or partitions.
The session store loses data. Workers cannot find recipients, so they fall back to push. Phones register again at their next heartbeat, within a minute.
The service dies between the save and the publish. The sender has the tick, but no worker is told. An online Ravi would wait. Fix: keep a small "not yet published" marker with the save, and let a sweeper publish what is left. Sync on reconnect also catches it.
Store writes slow down. Senders see a spinner and retry with the same client_msg_id. No duplicate is created.

What the interviewer asks next

A Mid answer sizes tens of thousands of messages a second, uses WebSockets behind a balancer, stores before the tick, puts a stream between sending and delivering, and handles offline users with storage plus push. A Senior answer adds seq with a single writer, sync as the real guarantee, at-least-once with duplicate removal, receipts as pointers, and reconnect storms.

Two follow-ups come often.

Multi-device. A phone and a laptop each hold their own connection, so a session entry lists devices. Each device keeps its own last-seq bookmarks. Read state belongs to the user, so a read_seq change is sent to the other devices as a receipt update.

End-to-end encryption. The phones hold the keys. The server stores only ciphertext, bytes that only the members can read. It can still assign seq, store, queue and push, because those use an envelope it can read: conversation, sender, seq and size. What changes? No server-side search or spam checks on text. Push alerts cannot show text built by the server. A new device cannot fetch old history readably from the server alone. In a group, the sender must encrypt for each member or use a shared group key, so cost grows with group size. Schemes differ, so name these trade-offs and offer to design the key exchange separately.

📌 Summary: Order comes from a per-conversation seq with one writer. Sync from the last seq is the real offline guarantee, and push is only a nudge. Store group messages once, and fan out only deliveries.

What Meera learns next

Every offline user got a push notification. But who sends it, what if the provider is slow, and how do we avoid sending it twice? Next: design a notification service for email, SMS and push.

Check your understanding

In a group chat, replies sometimes appear above the questions they answer. Messages are sorted by the time on each sender's phone. What is the best fix?
A gateway deploy drops its 50,000 phones, and all of them reconnect and sync in the same second. What should the design do?
Ravi was offline and got a push notification, but he missed it. Later he opens the app. Where does the missed message come from?
A group has 100 members. How should the design store a message sent to it?
The product adds end-to-end encryption. Which part of the design still works unchanged?