Chat app: build the design box by box

Part of the free System Design course on LogicWiz, in the chapter: Design a chat app (WhatsApp). The interactive whiteboard exercises and simulations run in the app.

Week 12: The tick that lied

"Asha saw one tick and closed the app. The server restarted a second later, and her message was gone. The tick had lied."

A support ticket Kabir keeps pinned above his desk

Version 1, and the number that breaks it

Start with the simplest design that could work. The phone sends each message with a normal HTTP call to one service. The service writes it to a database. The receiver's phone asks every 3 seconds, "anything new for me?"

It works for a demo. Then lesson one's numbers arrive. At peak, 10 million phones asking every 3 seconds is about 3.3 million requests per second, nearly all empty. We want the opposite: the server speaks when there is something to say.

The tool is a WebSocket. The phone makes one HTTP request that asks to upgrade, and after that the connection stays open in both directions. Either side can send at any moment. Holding a connection costs the server a little memory, and a well-tuned box can hold tens of thousands. Plan on 50,000 per box. So 10,000,000 / 50,000 = 200 gateway servers. A gateway here is a server that holds phone connections, checks who the user is, and forwards messages. Nothing more.

Draw it: phone, then a load balancer, then the gateways. The balancer picks a gateway once, when the phone connects. After that the connection stays on that gateway, so a simple TCP-level (layer 4) balancer can be enough.

Which server holds Ravi? The session store

Asha's connection sits on gateway 17. Ravi's sits on gateway 42. Gateway 17 cannot write to Ravi's socket. Somebody must know that Ravi is on gateway 42.

Meera's first thought: the load balancer will route each message to the right gateway. It cannot. It chose a gateway at connect time and knows nothing about messages. We need a session store: a fast in-memory store with one entry per user, "Ravi is on gateway 42".

  • When a phone connects, its gateway writes the entry with a TTL (time to live) of 60 seconds. The entry vanishes on its own after that.
  • Every 30 seconds the phone sends a heartbeat, a tiny "I am still here" message. Each heartbeat renews the TTL.
  • If a phone dies or loses signal, no heartbeats arrive, and the entry expires within a minute. No cleanup job is needed.
  • A user may have a phone and a laptop, so an entry holds a list of devices.

Every delivery asks this store one question, about 160,000 times a second at peak. Heartbeats add 10,000,000 / 30, about 333,000 renewals a second. That is roughly 500,000 operations a second: fine for memory spread over several nodes, and not for a disk database. Gateways keep no message state. So a gateway can be restarted at any time: its phones reconnect elsewhere.

Store first, then tick, then deliver

Now the part that makes the tick honest. A gateway passes Asha's message to the message service, which does five things in a fixed order.

  1. Drop duplicates. If this client_msg_id was seen, return the earlier result.
  2. Assign the seq for the conversation, say 4,812. One instance owns each conversation at a time, as the deep dives explain.
  3. Write the message to the store, in the conversation's partition.
  4. Acknowledge to Asha with the message id and seq. She sees one tick.
  5. Publish the message to a stream, keyed by conversation_id.

The order matters. The tick is a promise that the message is safe, so the write comes before it. Swap steps 3 and 4 and you rebuild the bug in the opening quote.

Why a stream (an ordered, replayable log of events) between the message service and the workers? Saving a message takes milliseconds. Delivering can be slow: look up members, find gateways, wait for a phone. If the message service did that itself, a slow phone would hold up the sender. The stream lets the service stop at the tick. A stream is split into partitions, and one key always goes to the same partition. Keyed by conversation_id, it keeps every conversation's messages in order. It also absorbs bursts: at peak, 80,000 messages a second become about 160,000 deliveries a second, and delivery workers catch up behind it.

Receipts follow the same road. When Ravi's app receives the message it sends an acknowledgement (an "I have it"). That moves delivered_seq up and travels back to Asha as a small message: two ticks. When Ravi opens the chat, read_seq moves up: blue ticks. Receipts are traffic too, so count them in the load.

One message from start to finish

sequenceDiagram
  participant A as Asha
  participant G1 as Gateway 17
  participant M as Message service
  participant S as Store
  participant K as Stream
  participant W as Worker
  participant R as Session store
  participant G2 as Gateway 42
  participant V as Ravi
  A->>G1: send
  G1->>M: message.send
  M->>S: save with seq
  M-->>G1: ack
  G1-->>A: one tick
  M->>K: publish
  K->>W: next message
  W->>R: where is Ravi
  R-->>W: gateway 42
  W->>G2: push
  G2->>V: message
  V-->>G2: ack
  G2-->>W: delivered
  W->>G1: receipt
  G1-->>A: two ticks

If Ravi is offline, the session store has no entry. The worker sends a push notification and stops. The message waits in the store until Ravi reconnects and asks for what he missed. For a group, the worker repeats the lookup for each member, up to 100. Reads are simple: opening a chat reads the newest 50 rows of one partition, and the inbox reads one user's partition.

Now build it. Start from a design that polls through one service.

Photos and videos stay out of the message path

A photo is about 1 MB. A text message is about 200 bytes. That is 5,000 times bigger. If photos travelled through gateways, the stream and the message store, they would crowd out every text.

So the phone uploads the file straight to object storage (a service for large files, with its own copies and a CDN in front). The upload uses a short-lived signed link from our service. The message then carries only the file's id and a small preview. Ravi's phone downloads the file from storage. The chat path keeps carrying small messages, and the large bytes take their own road.

📌 Summary: Open WebSockets let the server push. A session store tells workers which gateway holds each user. Store the message, then tick, then deliver. Media goes to object storage, with a link in the message.

What Meera learns next

The design runs end to end. Now Kabir asks the questions that separate answers: how is order kept inside a conversation, what exactly happens to an offline user, and what does a group of 100 cost? Next: the deep dives, and what breaks.

Check your understanding

  1. Worker W holds a message for Ravi, who is connected to gateway 42 of 200. How does W learn which gateway to call?
  2. A message service acknowledges Asha first and then writes to the store. It crashes between the two steps. What is the damage?
  3. A group member's phone is slow to accept messages. Why does the design put a stream between the message service and the delivery workers?
  4. Asha sends a 2 MB photo to a group. What is the right path for the file?