Notification service

A shared service that sends email, SMS and push for every team. It retries failures and never sends the same notification twice. The core ideas are a queue between accepting and sending, idempotency keys, backoff and a dead-letter queue.

Design a notification service that sends email, SMS and push, retries failures, and never sends the same notification twice.

Requirements

Functional

  • Internal services submit a notification (user, template, parameters, priority). The service picks channels from the user's preferences.
  • Send over email, SMS and push through external providers, with failover to a second provider for each channel.
  • Retry temporary failures. Never send the same notification twice, even if the caller submits it twice.
  • Respect user preferences: opt-outs per category, and quiet hours (no promotional SMS at 2 a.m.).
  • Report status per notification and channel (queued, sent, delivered, failed).

Non-functional

  • Scale: about 10 million notifications a day (about 115 a second) on average. Campaign and flash-sale peaks reach about 10,000 a second.
  • Latency: transactional messages (OTP, order confirmed) reach the provider within 2 seconds at p95, meaning 95 out of 100 messages. Promotional ones go within 15 minutes.
  • Priority isolation: a promotional blast to 5 million users must never delay an OTP.
  • Availability: 99.95% for accepting requests. Once a notification is accepted (HTTP 202, "accepted for processing"), it is never lost.
  • Respect provider rate limits, and send no duplicates from our side.

Out of scope

  • Template authoring UI and campaign segmentation (who gets a blast).
  • In-app inbox and its UI.
  • Running our own SMS gateway or mail servers; we use external providers.

Estimate

  • Average: 10M notifications a day / 86,400 s ≈ 115 a second; at about 1.5 channels each, ≈ 175 provider sends a second.
  • Peak: a flash-sale blast to 5M users spread over 10 minutes = 5,000,000 / 600 ≈ 8,300 a second; plan for 10,000 a second.
  • Workers: a provider call takes about 100 to 300 ms, so one worker thread does roughly 5 sends a second; 10,000 / 5 = 2,000 concurrent sends, e.g. 40 worker pods x 50 threads (provider rate limits will cap SMS lower; the queue absorbs the rest).
  • Writes: each notification is one insert plus about 2 status updates → at 10,000 a second that is 30,000 writes a second at peak, beyond one relational primary (comfortable at about 2,000) → a partitioned, write-friendly store.
  • Storage: about 1 KB per notification with its attempts → 10M x 1 KB = 10 GB a day; 90 days of history ≈ 900 GB, x3 replication ≈ 2.7 TB.
  • Backlog: if SMS providers allow about 1,000 a second in total and a 10-minute blast asks for 8,000 a second, the backlog grows by 7,000 a second to about 4.2M. Draining 4.8M SMS at 1,000 a second takes about 80 minutes. So a blast that size cannot meet the 15-minute target by SMS: send it mostly as push and email, or start the SMS earlier and stagger it.

API

  • POST /v1/notifications {idempotency_key, user_id, template_id, params, category, priority} -> 202 {notification_id, status: "queued"}
  • GET /v1/notifications/{id} -> 200 {status, channels: [{channel, provider, status, attempts, last_error}]}
  • POST /v1/notifications/batch {idempotency_key, user_ids[], template_id, params, category} -> 202 {batch_id}
  • PUT /v1/users/{id}/preferences {channels, opt_outs[], quiet_hours} -> 200
  • POST /v1/providers/{provider}/callbacks {provider_msg_id, status} -> 200 (delivery receipts from providers)

Data model

  • notifications(notification_id PK, idempotency_key UNIQUE per caller, user_id, template_id, params, category, priority, status, created_at): partitioned by notification_id.
  • deliveries(notification_id, channel, attempt, provider, provider_msg_id, status, error, ts): one row per attempt.
  • preferences(user_id PK, email, phone, push_tokens[], opt_outs[], quiet_hours, timezone): cached in Redis.
  • dedup keys (Redis): dedup:{caller}:{idempotency_key} -> notification_id, SET NX with a 24-hour TTL.
  • Topics: notify.{channel}.{high|low}, plus notify.retry (delayed) and notify.dlq.

A request, step by step

  1. The order service calls POST /notifications with idempotency_key "order-4821-confirmed", template order_confirmed and priority high. An idempotency key is an id chosen by the caller, so a repeated request is recognised and ignored.
  2. The API runs SET NX on the dedup key in Redis. SET NX writes the key only if it does not exist yet. If the key exists, the API returns the original notification_id with 202 and does nothing else. So a caller's retry never creates a second notification.
  3. The API loads the user's preferences from the cache and drops channels the user opted out of. It writes the notification row and publishes one message per channel to the high-priority topic, for example notify.sms.high and notify.push.high. It returns 202 in about 20 ms.
  4. A channel sender reads the message and checks the delivery row: has this channel already been sent? It takes a token from that provider's rate limit and calls the provider. Where the provider supports it, the sender passes our notification_id as the provider's reference or idempotency key.
  5. On success, the worker marks the channel "sent", and the provider's callback later marks it "delivered". On a timeout or a 5xx error (a failure on the server side), it records the attempt. It then publishes to the retry topic with the next backoff: wait 1, 2, then 4 seconds and so on. Each wait gets jitter, a small random delay so retries do not all fire together.
  6. The retry scheduler puts the message back on the queue when its backoff ends. After 5 failed attempts, or on a permanent error such as an invalid number, it parks the message in the dead-letter queue and raises an alert. A dead-letter queue is a holding area for messages that keep failing. The scheduler may also fail over to the second provider.

Deep dives

Never sending twice: where duplicates come from

Duplicates come from three places:

  • The caller retries the request.
  • The queue delivers a message again. Queues are at-least-once: every message arrives, sometimes more than once.
  • A worker crashes after the provider sent the message but before we recorded it.

Weak: hope the queue is exactly-once. Queues deliver messages again when a consumer crashes or when the consumer group rebalances (moves partitions between workers). So this fails in the first week.

Good: an idempotency key at the front door. The caller supplies one key per logical notification, such as "order-4821-confirmed". The API does SET NX in Redis, and the log also has a unique constraint on the key. A repeated request returns the original id. That removes duplicates caused by caller retries.

Great: idempotency at every hop. Each channel delivery has a stable id: notification_id plus channel. The worker checks and updates the delivery row with a conditional write, so it sends only if the status is "queued". Where the provider supports it, the worker passes that id as the provider's reference or idempotency key.

If a crash leaves the status unknown, the worker asks the provider for the status of that reference before resending.

Trade-off: true exactly-once delivery to a phone is impossible across an external provider. Suppose a provider accepts a send, the acknowledgement is lost, and the provider has no idempotency support. You must then choose between a possible duplicate and a possible miss. Say so, and choose per category: an OTP can repeat, but a "₹5,000 debited" alert should not.

Retries, backoff and the dead-letter queue

Weak: retry immediately in a loop inside the worker. When an SMS provider is struggling, 2,000 threads retrying at once turn a slowdown into an outage. They also block every other message behind them.

Good: exponential backoff with jitter, then a dead-letter queue. Exponential backoff means each wait is double the last one. Temporary errors are a timeout, HTTP 429 ("too many requests") or a 5xx server error. On one of these, wait 1, 2, 4, 8 and 16 seconds, each with random jitter.

Give up after about 5 attempts and move the message to a DLQ (dead-letter queue) that someone reviews. Permanent errors, such as an invalid number or an unsubscribed user, go straight to the DLQ without retries.

Great: retries off the main line, plus a circuit breaker per provider. Failed messages go to a separate retry topic with a due time. A scheduler puts them back on the main queue when they are due. So fresh OTPs are never stuck behind old failures.

Each provider has a circuit breaker: a switch that stops calls to a failing dependency for a while. When a provider's error rate crosses a threshold, workers stop calling it for 30 seconds and fail over to the secondary provider.

Every message also has an expiry. An OTP older than 5 minutes is useless, so it is dropped rather than retried.

Trade-off: you get more moving parts (retry topic, scheduler, breakers), and messages may arrive out of order. In return, one bad provider or one poison message (a message that fails every time) cannot stall everything.

Priorities: keeping OTPs fast during a blast

Weak: one queue for everything. Marketing schedules a Diwali blast to 5 million users. Those 5 million messages sit in the queue ahead of a login OTP. The OTP arrives 10 minutes late, and the user gives up.

Good: separate topics per priority, with dedicated workers. Use notify.sms.high and notify.sms.low, each with its own consumer group. High-priority workers are never busy with promotions. Admission control at the API rate-limits bulk callers, so a blast enters the system at a controlled pace.

Great: priority plus a budget per provider. Provider rate limits are shared by all traffic. So reserve a slice of each provider's throughput for transactional messages, say 30% of SMS capacity. Promotional traffic uses whatever is left.

Quiet hours and frequency caps (no more than 3 promotional pushes a day) are applied before enqueueing. Messages held for quiet hours go to the scheduler, not the hot queue.

Trade-off: the reserved capacity sits partly idle on a normal day, and promotional messages finish later during a peak. That is the right way round. Nobody complains that a sale SMS arrived 8 minutes late, but everyone notices a late OTP.

What breaks

  • An SMS provider degrades: its error rate and latency rise on its dashboard. Its circuit breaker opens, and traffic fails over to the secondary provider. Low-priority SMS backs up and drains later.
  • Kafka is unavailable: the API cannot enqueue, so it returns 503 (service unavailable). Callers retry with the same idempotency key, which is safe. Notifications already accepted into the log are sent again from the log when Kafka returns.
  • The Redis dedup data is lost: the unique constraint on idempotency_key in the log still catches duplicates, with higher latency.
  • A poison message, such as a template that always fails to render, reaches the DLQ after a few attempts instead of blocking a partition. An alert on DLQ size fires.
  • The backlog grows faster than it drains: alert on consumer lag and on the age of the oldest message per priority, not only on queue depth.

What each level is expected to cover

Mid level: A Mid candidate should put a queue between accepting and sending, and use workers per channel. They should retry with backoff, add a dead-letter queue and store status. They should use an idempotency key, so a resubmitted request is not sent twice.

Senior: A Senior candidate also traces every source of duplicates and makes each hop idempotent. They admit that exactly-once delivery through an external provider is impossible, and handle the unknown outcome. They isolate priorities so blasts cannot delay OTPs. They add per-provider rate limits, circuit breakers and failover, and apply preferences and quiet hours before sending.

Common mistakes

  • Sending synchronously inside the caller's request, so a slow provider stalls checkout.
  • Claiming the queue gives exactly-once delivery, and skipping idempotency.
  • Retrying instantly in a tight loop, with no backoff, jitter or retry limit.
  • One queue for OTPs and marketing, so a blast delays logins.
  • No dead-letter queue, so one bad message is retried forever.
  • Ignoring provider rate limits and user preferences (opt-outs, quiet hours).

Follow-up questions

  • The provider accepted the SMS, but our worker crashed before recording it. What happens next?
  • How would you send a Diwali offer to 5 million users in their local morning?
  • How do you add WhatsApp as a new channel without touching the calling services?
  • How would you measure delivery rate per provider and switch automatically?
  • What do you do with messages in the dead-letter queue?

Lessons behind this answer