Design it

Part of the free System Design course on LogicWiz, in the chapter: Design a notification service. The interactive whiteboard exercises and simulations run in the app.

Week 13, Day 3: The notification nobody knew about

"The API said 202 and went home. The server restarted a second later, and no log said that notification ever existed."

Kabir, reviewing Meera's first draft

Version one: call the provider and wait

Start with the simplest design, as you would on a whiteboard. The calling service sends a request. The notification service picks a channel, calls the provider and waits for the answer. Then it tells the caller "sent".

It works on a quiet day. Now use the numbers from lesson one. A provider call takes 200 ms, so a thread does 5 sends a second, and 10,000 sends a second need 2,000 threads. Then the provider has a bad minute and each call takes 4 seconds. A thread now does a quarter of a send a second, so the same load needs 10,000 times 4, which is 40,000 threads. The service runs out, callers time out, and checkout waits on a text message.

Meera's instinct: "The caller wants to know the message went out, so we must wait." Kabir's answer: the caller wants to know it will not be lost. That is a smaller promise, and we can keep it in 20 ms.

Version two: accept, save, queue, send

Split the job in two. The first half accepts the request and replies 202, which means "accepted for processing", not "sent". The second half sends later, at its own pace. A queue, a waiting line that holds messages safely, sits between them.

Each box is earned by a number.

  • Gateway. Checks which service is calling and limits each caller, so one team's bug cannot flood everyone.
  • Notification API. Four stateless copies. It validates, deduplicates, applies preferences, saves and enqueues.
  • Redis cache. The API reads preferences and checks the dedup key on every request, up to 10,000 a second. A cache answers in under a millisecond. Run three copies, so one failure does not push that load onto the preferences database.
  • Notification log. At 30,000 writes a second it needs a store partitioned by notification id, such as Cassandra, with three copies. The trade-off: ad hoc queries ("all failed SMS in the last hour") get hard, so dashboards read metrics instead.
  • Queue. The peak is 10,000 sends a second and the SMS provider takes 1,000. Something must hold the difference. One topic per channel and priority, for example notify.sms.high.
  • Channel senders. 40 worker processes with 50 threads each, about 2,000 sends in flight. They call the providers, with a second provider per channel as backup.
  • Monitoring. Queue lag and the age of the oldest message, per priority, and each provider's error rate.
flowchart LR
  C["Calling services"] --> G["Gateway"]
  G --> A["Notification API"]
  A --> R["Redis: dedup + preferences"]
  A --> L["Notification log"]
  A -->|"enqueue"| Q["Queues by channel + priority"]
  Q --> W["Channel senders"]
  W --> P["Providers"]
  W --> L
  P -.->|"receipts"| A

The order of steps inside the API matters. Check the dedup key. Load preferences. Save the log row, publish to the queue, and only then reply 202. Suppose the server dies between saving and publishing. A sweeper, a small job that runs every minute, finds rows still "queued" for more than a minute and publishes them. It may publish a message twice, which the sender's conditional write (deep dive one) absorbs. The reverse order, publish first, can send a message that has no log row, so nobody can see its status.

One request in, many deliveries out

One request for Rohan, with three channels allowed, becomes three messages: one per channel. Each has a stable delivery id, the notification id plus the channel name, and goes to its own topic, such as notify.push.high and notify.sms.high. Each channel can then succeed, fail and retry on its own. If the push works, the SMS backup can be cancelled.

A batch is the same idea, one level up. A request for 5 million users must not create 5 million messages before replying. The API stores one batch row and replies 202. An expander, a worker that reads the recipients in pages, creates the per-user messages at a controlled pace. This is fan-out: one input, many outputs.

The message on the queue holds a template id, a template version and the parameters, not finished text. The sender renders it at send time, in the user's language and the channel's format: short for SMS, richer for email. The benefit: a typo fixed while 2 million messages wait applies to all of them. The cost: a bad template edit also reaches queued messages, so templates are versioned and an OTP template version is pinned.

Follow one OTP through

sequenceDiagram
  participant S as Login service
  participant A as Notification API
  participant Q as Queue
  participant W as SMS sender
  participant P as SMS provider
  S->>A: POST (key otp-9981, high)
  A->>A: dedup, preferences, save
  A->>Q: notify.sms.high
  A-->>S: 202 in about 20 ms
  Q->>W: message
  W->>P: send (reference id)
  P-->>A: receipt: delivered

The login service posts with the key otp-9981. The API sets the dedup key, finds that Rohan allows SMS, saves the row and publishes. It replies in about 20 ms. A sender takes the message, checks that this delivery is still "queued", renders the template, takes a token from the provider's rate limit and calls the provider. The provider replies "accepted" and later posts a receipt. A read, GET /v1/notifications/{id}, returns the status per channel from the log.

Delivery tracking: sent is not delivered

A delivery moves through queued, sent, delivered or failed. "Sent" means a provider accepted it. "Delivered" comes from a receipt, which can be late, arrive twice, or never arrive for SMS.

So the log lets a status move only forward. If "delivered" arrives before our worker records "sent", the final answer stays "delivered". The callback endpoint checks the provider's signature where the provider offers one, since anyone could post to it. Every attempt is one row in deliveries, with the provider, the reference and the error, so support can answer "why did I not get it?".

Now build the design on the board.

📌 Summary: Accept, save, queue, then send. Reply 202 only after the row is saved and the message is queued. One request becomes one message per channel. Status moves forward only.

What Meera learns next

The design works on a normal day. Interviewers push on three things: how you never send twice, how you retry without a storm, and how an OTP escapes a sale blast. Those are the deep dives next.

Check your understanding

  1. The API replies 202 for an OTP, and the server restarts before sending. Which choice decides whether the OTP is lost?
  2. Marketing fixes a typo in a template while 2 million messages wait in the queue. Why do those messages carry a template id, not finished text?
  3. The log needs 30,000 writes a second at peak, and one relational primary is comfortable at about 2,000. What is the right response?
  4. A delivery receipt saying "delivered" reaches the API before the worker has recorded "sent". What should the log do?
  5. Marketing posts one batch for 5 million users. Where should the per-user messages be created?