Deep dives, failures and follow-ups

Part of the free System Design course on LogicWiz, in the chapter: Design a notification service. The interactive whiteboard exercises and simulations run in the app.

Week 13, Day 5: The OTP that arrived at 9:11

"Marketing sent the Diwali offer to 5 million people at 9 a.m. The login codes sat behind it. By 9:11, users had given up."

From an incident review at SwiftKart

You have the design. Now the interviewer picks one or two hard parts and pushes. There are three. For each, give a weak option, a good one, a great one, and the trade-off.

Deep dive one: never sending twice

Duplicates have three sources. The caller retries a request. The queue delivers a message again after a worker crashes, because most queues promise at-least-once: every message arrives, sometimes twice. Or a worker crashes after the provider sent the text but before we wrote that down.

Weak: "The queue is exactly-once." Some queues offer that only inside their own system. A crash before the acknowledgement still makes the queue send the message again, and a call to an outside provider is not part of the guarantee.
Good: an idempotency key at the front door. The caller sends "order-4821-confirmed". The API runs SET NX in Redis, which writes the key only if it does not exist yet, with a 24-hour expiry. The log also enforces uniqueness on the key, for example by deriving the notification id from the caller and key and inserting only if absent. So if Redis loses its data, or the 24 hours pass, the log still catches the repeat. If the same key arrives with different content, return an error, because that is a caller bug. The key stops one request from running twice; it cannot stop a caller from inventing a fresh key for the same event. If the key exists but the log has no row, the first attempt died halfway: finish its work with the same id.
Great: idempotency at every hop. A sender updates the delivery row with a conditional write: set "sent" only if the status is "queued". Two workers cannot both win. It passes the delivery id to the provider as the reference, so a provider that supports it ignores a repeat.

One gap remains. The provider accepted the send, our worker died before recording it, and the provider has no reference support. Now you must choose between a possible duplicate and a possible miss. Say so, and choose by category. A repeated OTP is a nuisance, not a harm. A repeated "₹5,000 debited" alert is not, so first ask the provider for the status of that reference.

Deep dive two: retries without a storm

Temporary errors are a timeout, a 429 (too many requests) or a 5xx (a failure on the provider's side). If a 429 carries a Retry-After header, wait at least that long.

Weak: retry at once in a loop inside the worker. When the SMS provider struggles, 2,000 threads hammer it, and they block every message behind them.
Good: exponential backoff with jitter, then a dead-letter queue. Allow 5 attempts. That is 4 waits: 1, 2, 4 and 8 seconds, about 15 seconds in all. Jitter adds a random part to each wait. Without it, 1,000 messages that failed together retry together, again and again. After the fifth failure the message moves to the dead-letter queue (DLQ), a holding area for messages that could not be sent. Permanent errors, such as an invalid number or an unsubscribed user, go straight there with no retries.
Great: retries off the main line, a breaker per provider, and an expiry. Failed messages go to a retry topic with a due time, and a scheduler puts them back when due. Fresh OTPs never wait behind old failures. A circuit breaker stops calls to a failing provider for a while, say 30 seconds, once its error rate crosses a threshold, and senders fail over to the second provider. An OTP older than its validity, say 5 minutes, is useless, so drop it. For an OTP, skip the long waits and fail over after the first error, because the 2-second target allows nothing more.

The trade-off: more parts, and messages can arrive out of order. In return, one bad provider or one poison message, one that fails every time, cannot stall everything. A DLQ is useful only when someone owns it: alert on its size, inspect, fix, replay.

Deep dive three: lanes and provider budgets

Weak: one queue for everything. Marketing's 5 million messages sit ahead of the OTP.
Good: separate topics and workers by priority, notify.sms.high and notify.sms.low. High-priority workers never touch promotions. The API also limits bulk callers, so a blast enters at a controlled pace.
Great: lanes plus a budget per provider. The provider's limit, 1,000 SMS a second, is shared by all your traffic. Enforce it with a token bucket (the limiter idea from the entry layer, pointed outward), kept in Redis so all 40 workers share one count. If the provider answers 429, slow down. Reserve a slice for transactional traffic: 30% is 300 a second that promotions cannot touch, so promotions get 700.

Do the sum. The blast is 4,800,000 SMS. At 700 a second that takes 4,800,000 divided by 700, about 6,900 seconds, or 114 minutes. With the full 1,000 it was 80 minutes, so the reserve costs the blast 34 minutes. The OTP lane stays at 300 a second and is never delayed. That is the right price, and it is also why big blasts go mostly by push and email.

Apply preferences, quiet hours and frequency caps (no more than 3 promotional pushes a day) before enqueueing. A message held for quiet hours goes to the scheduler, not the hot queue. A message that waits 100 minutes may meet a user who opted out meanwhile, so low-priority senders re-check the opt-out flag, a cached read, just before sending.

The trade-off: on a normal day part of the reserve sits idle, and promotions finish later at a peak. Nobody complains that a sale text came 8 minutes late. Everyone notices a late OTP.

Draw the lanes and the retry path.

What breaks, and how you notice

A provider degrades. Its error rate and latency rise. The breaker opens and traffic fails over. Low-priority SMS backs up and drains later.
The queue is unavailable. The API cannot enqueue, so it replies 503. Callers retry with the same key, which is safe. Rows already saved are republished by the sweeper.
Redis loses the dedup data. The unique constraint in the log still catches duplicates, with higher latency.
A poison message, such as a template that always fails to render, reaches the DLQ instead of blocking a partition. Alert on DLQ size.
The backlog grows faster than it drains. Alert on consumer lag and the age of the oldest message per priority, not on queue depth alone.

Mid, Senior and the follow-ups

A Mid answer: "Callers post to an API. I queue per channel, workers send with retries and a DLQ, and an idempotency key stops repeats." That is a sound base. A Senior answer adds: "Duplicates come from three places, so every hop is idempotent, and after a provider timeout the outcome is unknown, so I choose by category. OTPs get their own lane and a reserved slice of provider capacity. Blasts are metered and go mostly by push and email."

Expect these follow-ups.

"The provider accepted the SMS, but our worker crashed before recording it." Ask the provider for the status of our reference. If it has none, choose duplicate or miss by category.
"How do you add WhatsApp as a channel?" A new topic, a new sender and a preferences field. Callers send a template and a category, so they do not change.
"What goes in the DLQ, and then?" Alert, inspect, fix the cause, replay. Invalid numbers are marked failed and the address is removed.

📌 Summary: Make every hop idempotent and admit the unknown outcome. Retry with backoff, jitter and a dead-letter queue. Give OTPs their own lane and a reserved share of each provider's limit.

What Meera learns next

A notification service sends one message to one phone and cares about delivery. The next question, the news feed, shows one post to thousands of followers' screens. You must choose between writing the post into every feed when it is posted, and building the feed when each follower opens the app. The fan-out you met in the 5-million blast, and the in-app inbox, come back as the whole design.

Check your understanding

A worker tells the SMS provider to send, the provider accepts, and the worker crashes before recording it. The queue redelivers the message. Which step best prevents a duplicate text?
A sender makes at most 5 attempts and waits 1 second, then doubles the wait each time. How many waits happen, and how long in total before the dead-letter queue?
A provider has a blip, and 1,000 messages fail in the same second. Why add jitter to the backoff?
The SMS provider allows 1,000 a second. SwiftKart reserves 30% for transactional traffic and sends a 4.8 million SMS blast. What happens?
A message to a phone number that does not exist fails with "invalid number". What should the sender do?