Queues: a buffer between fast and slow

Part of the free System Design course on LogicWiz, in the chapter: Async systems. The interactive whiteboard exercises and simulations run in the app.

Week 7: The SMS provider that froze checkout

"Orders were failing, and the payment service was perfectly healthy. The cause was the thing that sends a text message."

Meera, on the incident call

One slow call, and checkout is dead

During a festival sale, SwiftKart's checkout did four things in a row: charge the payment, reserve stock, create the order row, send a confirmation SMS. All four were synchronous, meaning each call waited for the answer before moving on. Then the SMS provider got slow. Instead of answering in 50 milliseconds it took 4 seconds.

Look at what that does. The checkout service has 200 worker threads. Each thread is now stuck for about 4 seconds on one order. So the service can finish only 200 / 4 = 50 orders per second. The festival peak is 500 per second. Threads run out within a second, new requests wait, then time out, and customers see "something went wrong" for an order whose payment already succeeded.

A text message is not even essential to placing an order. Yet the slowest step set the speed of everything.

💡 Tip: In a chain of synchronous calls, if failures are independent, your availability is roughly the product of everyone's, and your latency is the sum. One slow link drags the whole chain.

A queue: a waiting line that keeps messages safe

A queue is a place where one part of the system drops a message ("send an SMS for order 4821") and another part picks it up later. The one who drops it is the producer. The ones who pick up and do the work are consumers, often called workers. The queue holds messages safely in between.

Now checkout becomes: charge, reserve, create the order, drop one message on the queue, reply "order placed" to the customer. Dropping a message takes a couple of milliseconds. The slow SMS provider is now the workers' problem, not the customer's.

The arithmetic of a backlog

A queue is a buffer between something fast and something slow. Real numbers show how much it buys you. Suppose the SMS workers can send 200 messages per second in total (20 workers, about 10 each).

  • The festival burst: 500 orders per second for 60 seconds. That is 30,000 messages arriving.
  • During that minute the workers clear 200 per second, so the backlog grows by 300 per second. After 60 seconds: 18,000 messages waiting.
  • Afterwards traffic falls to a normal 100 per second. Workers still clear 200, so the backlog shrinks by 100 per second.
  • Drain time: 18,000 / 100 = 180 seconds, three minutes.

Customers who ordered at the peak get their SMS up to a few minutes late. Nobody saw an error. Compare that with the synchronous version where people lost orders. Want it faster? Double the workers to 400 per second and the burst backlog is only (500 - 400) x 60 = 6,000 messages, gone about 20 seconds after traffic drops.

Check the arithmetic yourself below. Set a burst of 500 a second against 20 workers at 10 each. Read the peak backlog and drain time off the chart. Then add workers until the backlog barely rises. Finally drop the workers below the normal arrival rate and watch the line never come back down.

Meera's first thought was that the queue made the system faster. It does not do less work; it moves work out of the customer's wait. And the buffer has a limit. If workers are permanently slower than arrivals, the backlog grows forever. Watch queue depth and the age of the oldest message, and alert on them.

Retries, backoff and the dead-letter queue

Workers fail. The SMS provider returns an error, a network call times out. The usual answer is to retry, but carelessly. If 20 workers all retry instantly against a provider that is struggling, you add load to a system that is already overloaded.

Use exponential backoff with jitter, as in Week 3. Wait 1 second, then 2, then 4, then 8. Add a small random extra (the jitter) to each wait so workers do not all retry at the same moment. After a fixed number of attempts, say 5, stop.

Some messages will never succeed: a malformed phone number, a bug. Retrying forever blocks the line. So after the last attempt the message moves to a dead-letter queue (DLQ), a side queue for messages that could not be processed. A person or a script looks at it later. A DLQ with messages in it is a signal to act on. If nobody watches it, failed orders sit there and nobody notices.

⚠️ Warning: Retries mean the same message can be handled twice. Next lesson is about making that safe.

Queue versus stream, in plain words

A queue is a to-do list: once a worker finishes a message, it is gone, and each message goes to one worker. Good for "do this job".

A stream is an append-only log, like a diary that is never erased for a while. Many different readers can each read it from their own bookmark, and a new reader can start from yesterday. Good for "many systems care about what happened" and for replaying history.

For the SMS job, a queue is right. For "order placed" that billing, analytics and notifications all want, a stream fits better.

flowchart LR
  A["Checkout"] --> Q["Queue"]
  Q --> W["SMS workers"]
  W --> P["SMS provider"]
  W -. "after 5 failures" .-> D["Dead-letter queue"]

📌 Summary: A synchronous chain is as slow as its slowest link. A queue moves slow work out of the customer's wait and buffers bursts. Retry with backoff and jitter, then dead-letter.

What Meera learns next

Retries mean a message may arrive twice. Next: what "delivered" really promises, why exactly-once is a myth, and the patterns that stop a double charge.

Check your understanding

  1. A sale sends 600 SMS messages a second for 60 seconds. Workers clear 400 a second in total. Afterwards arrivals fall to 100 a second. How big does the backlog get, and how long does it take to drain?
  2. After adding a queue, the SMS workers clear 150 messages a second, but orders arrive at a steady 200 a second all evening. What happens?
  3. The SMS provider is struggling. Twenty workers retry every failed send immediately, with no limit. What should change?
  4. Billing, analytics and notifications all need every "order placed" event, and analytics wants to replay last week after a bug fix. What should checkout publish to?