Deep dives: two taps, late payments and the 10:00 rush

Part of the free System Design course on LogicWiz, in the chapter: Design ticket booking (BookMyShow). The interactive whiteboard exercises and simulations run in the app.

Week 17: Two taps, one seat

"Both taps reached the server in the same millisecond. Both saw H7 free. Both got the hold."

Meera, tracing a bug in her first version

Never two people in one seat

Weak: read, then write. The code reads the seat, sees FREE, and then updates it to HELD. Two requests can both read FREE before either writes. Both then write. Trusting the cached map makes it worse.

Good: one guarded update, in one transaction.

UPDATE seats SET status = 'HELD', hold_id = :hold, held_until = now() + 8 minutes
WHERE show_id = :show AND seat_id IN (:seats)
  AND (status = 'FREE' OR (status = 'HELD' AND held_until < now()))

If the number of changed rows equals the number of seats asked for, commit. If not, roll back all of it and answer 409. The database makes the second writer wait for the first, then checks the condition again against the new value. The second finds HELD and changes zero rows. Exactly as many requests succeed as there are seats.

You can also lock the rows first, with a row lock (the database makes others wait on those rows) and then update. For "any two seats" you can lock free seats and skip locked ones. Lock seats in a fixed order, such as sorted by id. If one customer wants H7 then H8 and another wants H8 then H7, mixed order creates a deadlock, where each waits on the other until the database cancels one of them.

A candidate in a mock said, "I will put a distributed lock around the seat." That adds a second place to be wrong. A lock held by a paused process can expire while its owner still thinks it holds it. Keep the rule in the database that stores the seat.

Great: shard by show, and queue the hottest shows. Shard (split) the database by show, so a hot show shares no locks with a quiet one. If each hold transaction keeps its locks for about 1 to 2 milliseconds, one hot show can do roughly 500 to 1,000 holds a second. If 50,000 people chase 90 seats and lock waits start to time out, send that show's holds through a queue keyed by show, with one worker handling them in order. There are no lock waits. Trade-off: one more hop and a worker to keep alive, so use it only for the hottest shows. We repeat the rule from Day 1: overbooking is not allowed, and this update makes it impossible.

A hold that expires, and a payment that does not care

The timeline: the hold starts at 0:00, the payment session ends at 7:00, the hold ends at 8:00.

Weak: a payment window longer than the hold. A customer pays at minute 9. The hold expired at 8, the seat was sold to someone else, and the first customer is charged for nothing.

Good: payment shorter than the hold, and a guarded confirmation. With 7 against 8, a payment cannot normally finish after the hold ends. On the webhook, confirm with another guarded update: set BOOKED only where the seat is HELD by this hold. Make it idempotent (running it twice has the same result as once). The gateway retries webhooks, so the booking table has a unique payment_ref. A second delivery finds the booking already there and returns it. One payment, one booking.

Great: also handle the late success. A bank may confirm after 8 minutes. The guarded update finds the hold EXPIRED, or the seats already taken. The service then tries to hold the same seats again. If they are still FREE, it books them. If someone else took them, it refunds automatically and tells the customer. A reconciliation job, every few minutes, compares payments with bookings and asks the gateway about payments that are still pending. Before the sweeper frees a hold, it checks for a pending payment on it.

Trade-off: the 7-minute limit makes a few slow UPI approvals fail. The alternative is charging people for seats they did not get.

Now write your own plan for the hardest part.

The 10:00 rush: a waiting room

At 10:00 the edge sees about 100,000 requests a second, for 270,000 seats. The database can serve about 1,000 holds a second on one hot show.

Weak: autoscale. Autoscaling reacts in minutes and the rush lasts seconds. The database cannot be scaled out in seconds. Extra servers only send more competing requests to the same rows. Users time out, retry, and multiply the load.

Good: rate limit and return 429. This protects the backend, but getting in becomes a lottery. People learn to keep pressing refresh, so the storm grows, and bots win.

Great: a waiting room. People join before 10:00 and get a signed token, with their position kept in a sorted set in a fast store. They watch a static page from the CDN, which is cheap even at 100,000 requests a second. At 10:00 the room lets people in, in order, at about 3,000 a second, a rate chosen from load tests. Each admitted person gets a short-lived admission token, and the gateway rejects booking calls without it. Each person spends about 60 seconds choosing seats, so about 3,000 times 60 = 180,000 sessions are in flight. That is easy for stateless booking servers.

Trade-off: some people wait a minute or two and then learn the seats are gone. That honest answer is far better than 30 minutes of errors.

What breaks, and how you notice

  • Expiry queue falls behind. Seats still stop counting after held_until, so users are not blocked. Alert on the age of the oldest unfired expiry.
  • Webhooks are late. Alert when the time from payment to booking passes 2 seconds at p99. Reconciliation repairs the rest.
  • A shard primary fails. Holds on those shows return 503 for a moment while a copy takes over. The waiting room admits fewer people for them.
  • The waiting-room store fails. Tokens are signed, so admitted users continue. New users see "retry in a minute" from the CDN.

Also watch the hold success rate, the 409 rate and the admission rate.

What Mid and Senior answers sound like

A Mid answer uses a guarded update or row lock, adds holds with an expiry, makes the webhook idempotent, and sees that 10:00 needs admission control. A Senior answer also sizes the waiting room from numbers, shards by show and queues the hottest show. It lines up the 7 and 8 minute timeouts and refunds late payments. It says exactly where the system is strongly consistent and where it shows hints.

Expect follow-ups.

  • Bots with thousands of accounts? Allow one active session per user, limit joins per phone number and device, and verify before joining the room.
  • Best available two seats together? Query free neighbouring seats in one row, then run the same guarded update on them.
  • How do you test the rush? Replay synthetic users in staging against one hot show, and watch lock waits and 409 rate.

📌 Summary: One guarded update decides every seat. Keep the payment window shorter than the hold, confirm idempotently, and refund late payments. Admit the 10:00 crowd through a waiting room, at the rate the database can serve.

What Meera learns next

Tickets are tiny rows. Next question: files that are gigabytes in size, saved on a laptop and appearing on a phone. File storage and sync: chunks, deduplication and conflicts.

Check your understanding

  1. Two taps on H7 reach the booking service in the same millisecond. The code reads the seat, sees FREE, then updates it. Both customers get a hold. What is the best fix?
  2. A payment webhook arrives at minute 9. The hold expired at minute 8 and H7 is now held by someone else. What should the system do?
  3. The payment gateway delivers the same "paid" webhook three times because our reply was slow. What makes sure there is only one booking?
  4. At 10:00 the edge sees 100,000 requests a second and the database serves about 1,000 holds a second on a hot show. Which approach fits?
  5. One show has 50,000 people for 90 seats, and the hold updates time out on row-lock waits. What do you add for that show?