Delivered twice, charged once

Part of the free System Design course on LogicWiz, in the chapter: Async systems. The interactive whiteboard exercises and simulations run in the app.

Week 7, Day 4: The customer charged twice

"The queue delivered the payment message twice. We did exactly what it told us. Both times."

Meera, reading the postmortem

Exactly-once delivery is a myth, and here is why

A worker takes a "charge Rohan 480 rupees" message, calls the payment service, and the payment succeeds. Just before the worker can tell the queue "done", the worker's machine crashes. The queue never hears back. What should it do? It cannot tell "the worker died before charging" from "the worker charged and died before acknowledging". Both look the same from outside.

It has two choices. Redeliver the message: it might charge twice. Or drop it: it might never charge at all. These are at-least-once delivery (a message arrives one or more times) and at-most-once (zero or one time). The same trap sits between any two machines separated by a network: a lost reply is indistinguishable from a lost request.

The team had assumed a good queue guarantees exactly-once delivery. Exactly-once delivery across a network is impossible. Exactly-once effects are achievable: you deduplicate at-least-once delivery, as the next section shows. For payments the safe default is at-least-once, because a late charge beats a lost one.

Idempotent consumers: make the second time harmless

If duplicates are unavoidable, make them harmless. An operation is idempotent if doing it twice has the same result as doing it once. "Set the order status to PAID" is idempotent. "Add 480 to the wallet" is not.

The method: give every message a unique idempotency key, for example the order id plus the step name. Say the work is a write to the consumer's own database. The consumer records the key in a table with a unique constraint, in the same transaction as the work. If the key already exists, it skips the work and acknowledges. If the work is a call to another system (a payment provider, an SMS gateway), pass the key along as that call's idempotency key so the other side deduplicates too. Payment APIs commonly work this way: a retry carrying the same reference returns the original result instead of debiting again. On the client side, the key is a UUID (a random unique id) generated once per checkout and reused on every retry, as in Week 4. Between services, it is the order id plus the step name.

At-least-once delivery plus an idempotent consumer gives effectively-once. That is the phrase to use.

The outbox: never save the order but lose the message

Another bug lives on the producer side. The order service does two things: write the order to its database, then publish "order placed" to the queue. These are two different systems, so no single transaction covers both. Crash between them and you have an order nobody hears about. Reverse the order and you announce an order that was never saved.

The transactional outbox fixes this. Inside the same database transaction that saves the order, also insert a row into an "outbox" table describing the message. Either both rows are saved or neither is. A separate small process then reads unsent outbox rows, publishes them to the queue, and marks them sent. If it crashes after publishing but before marking, it publishes again, which is fine because consumers are idempotent.

💡 Tip: Outbox plus idempotent consumer is a pair. The outbox guarantees "never lost"; idempotency makes "sometimes twice" safe.

A saga: a long order with no big transaction

Placing an order touches three services, each with its own database: payment, inventory, delivery. You cannot wrap them in one transaction. A saga is a sequence of local steps, each with a compensation, an action that undoes it:

1. Charge payment (compensation: refund)
2. Reserve stock (compensation: release stock)
3. Assign a rider (compensation: cancel assignment)

If step 3 fails because no rider is free, the saga runs the compensations backwards: release the stock, refund the payment. The customer sees a refund, not a half-finished order. A compensation is a new action, not a rollback: the customer may briefly see the charge, and a refund can fail and must be retried. Compensations are themselves messages, so they must also be idempotent.

Fan-out: do the work at write time or at read time

SwiftKart sends a "weekend deal" from a brand to its 2 million followers, shown in a notification feed.

Fan-out on write: when the brand posts, immediately write one feed entry per follower: 2 million writes, done by workers in the background. Opening the feed afterwards is a single cheap read. Fast to read, expensive to write.

Fan-out on read: store the post once. When a user opens the feed, fetch recent posts from the roughly 200 brands they follow and merge. Cheap to write, heavier to read.

The numbers decide. With 2 million followers, one post costs 2 million writes. At 20,000 writes per second of background capacity, the last follower sees it after about 100 seconds. If a brand posts once a day but followers open the app ten times a day, paying at write time wins easily. Now take a brand with tens of millions of followers who mostly never open the app. Writing 50 million feed entries is wasteful, so merge that brand's posts at read time instead. Following thousands of accounts, by contrast, makes read-time merging expensive, since every open means thousands of lookups.

Most feeds use a hybrid: pre-compute for ordinary accounts, and for the huge ones merge at read time. Pick by asking which side happens more often and which can tolerate delay.

📌 Summary: Exactly-once delivery is impossible, but idempotent consumers give exactly-once effects. An outbox ensures a saved change is always announced. A saga undoes failed multi-service steps with compensations.

What Meera learns next

Messages are safe now, but copies of data on different machines can disagree. Next week: consistency, quorums and why "what the user sees" depends on which replica answers.

Check your understanding

1. A worker charges Rohan's card for an order, then crashes before acknowledging the message, so the queue redelivers it. What actually prevents a second charge?
2. A consumer may receive the same message twice. Which of these actions is safe to run twice without any extra checks?
3. The order service saves an order, then publishes "order placed" to the queue. It crashed between the two, and the order was never announced. What is the fix?
4. In the order saga, payment is charged and stock reserved, then no rider is free. What should happen?