Designing for failure

Part of the free System Design course on LogicWiz, in the chapter: Reliability and observability. The interactive whiteboard exercises and simulations run in the app.

Week 9: The night the recommendations box took down checkout

"Recommendations were down. Fine, nobody needs them. So why could nobody pay?"

Meera, 9:14 pm on a sale Saturday

Availability is a number of minutes, not a feeling

Saturday night, a festival sale. SwiftKart's recommendations service ("people also bought") got slow. Not broken, but slow: answers that normally took 50 milliseconds now took 30 seconds. Four minutes later, checkout stopped working too. Meera spent the night learning one lesson. In a distributed system, failure spreads unless you design it to stop.

Start with the word everyone uses loosely. Availability is the share of time (or of requests) the system does its job. Turn the percentage into minutes and it stops being abstract:

99%: about 3.65 days of downtime a year.
99.9% ("three nines"): about 8.8 hours a year, or 43 minutes a month.
99.95%: about 4.4 hours a year.
99.99% ("four nines"): about 53 minutes a year, or roughly 4 minutes a month.

Each extra nine cuts the allowed downtime tenfold. It usually costs far more than the last one. Four minutes a month is too short for a person to wake up, open a laptop and read an alert. Four nines means machines must detect the failure and recover on their own.

Pick a target in the visualiser and read off the downtime it allows. Then build a chain of three parts, watch the total fall, and give the weakest part a second replica.

Parts in a chain multiply; replicas help only if they fail alone

Checkout needs the load balancer, the checkout service and the orders database, all at the same moment. If any one is down, checkout is down. When parts are in series, their availabilities multiply:

Chain availability = A1 × A2 × A3

Three parts at 99.9% each: 0.999 × 0.999 × 0.999 ≈ 0.997, so about 99.7%. That is roughly 26 hours of downtime a year, three times what any single part allows. A synchronous dependency is a service you must wait for before you can answer. Every one you add (payments, coupons, recommendations) puts one more number below 1 into the product.

Redundancy works the other way. Suppose you run two copies of a part, and one copy is enough. The pair fails only when both are down at once. If each copy is 99% available and their failures are independent, both are down 1% × 1% = 0.01% of the time, so the pair is 99.99%.

Meera's first reaction was: "So two replicas give us four nines." Kabir's answer: only if they fail independently. Picture two database copies in the same rack, on the same power supply, receiving the same bad config change. They go down together, and the pair is barely better than one. Real redundancy means different machines and different zones (separate data centres). It also means rolling out changes to one copy at a time.

A single point of failure is any part of the chain with no redundant copy: the lone database, the lone load balancer, the one person who knows how to deploy.

Timeouts, retries and the retry storm

Back to Saturday. The checkout page called recommendations with no timeout (no limit on how long to wait for an answer). Each checkout server had 200 worker threads and took about 100 requests a second. With every request now waiting 30 seconds, it needed 100 × 30 = 3,000 threads. It had 200. In about two seconds every thread was stuck waiting on recommendations, and payments queued behind them.

Rule one: every network call gets a timeout. Choose it from the callee's normal speed (the callee is the service being called). Recommendations usually answer in 50 ms, and 99 calls in 100 finish within 150 ms. So a 300 ms timeout is generous. Without a timeout, a slow dependency is worse than a dead one. A dead one at least fails fast.

Rule two: retry with care. In Week 3 you met exponential backoff with jitter: wait, retry, wait twice as long, and add randomness so clients do not return at the same moment. Retries fix brief blips. They hurt when the callee is overloaded, because they add load exactly when it can least take it. They also multiply through layers. Suppose the app, the gateway and the checkout service each make up to 3 attempts. One tap can become 3 × 3 × 3 = 27 calls to the database. That is a retry storm. Retry at one layer, cap the attempts, and retry only operations that are safe to repeat (idempotent, from Week 7).

⚠️ Warning: A retry without backoff is a load test you did not schedule. Ten thousand clients retrying instantly against a struggling service keep it down.

Circuit breakers, bulkheads and graceful degradation

A circuit breaker is a switch in front of a dependency. While calls succeed it is closed and lets them through. Say more than half of the last 20 calls fail or time out. Then it opens: for the next 30 seconds every call fails immediately, without touching the dependency. The caller gets an instant answer instead of a 300 ms wait, and the sick service gets room to recover. Then the breaker goes half-open and lets a few trial calls through. If they succeed, it closes again.

A bulkhead gives each dependency its own small pool of threads. The name comes from the walls inside a ship that stop one flooded compartment from sinking the whole ship. Recommendations get at most 20 of the 200 threads; payments get their own. When recommendations hang, they can trap 20 threads, never all 200.

Graceful degradation decides what the user sees when a part fails. Recommendations down? Hide the box. Prices slow? Show cached prices from a few minutes ago, and re-check the price at payment. Only the parts that must be right are allowed to fail the request: payment and creating the order. For each dependency, finish the sentence: "If this is down, we show ______."

When one slow call in a hundred becomes your normal

Fan-out means one request calls many services at once. Fan-out makes rare slowness common. The SwiftKart home page calls about 20 services: banners, offers, cart count, delivery slots and more. Suppose each call is slow (over a second) only 1% of the time. The page is slow if any call is slow:

Chance a request hits a slow call = 1 − (1 − p)^n

With p = 1% and n = 20: 1 − 0.99^20 ≈ 18%. Nearly one page in five is slow, although every service is "99% fast". At 100 calls it is about 63%. The tail is the slowest few percent of requests, and with fan-out many users land in it.

One fix is a hedged request. If a call has not answered by its usual 95th-percentile time (the time 95 calls in 100 beat), send a second copy to another replica. Take whichever answers first. That costs roughly 5% extra calls and cuts the tail sharply. Hedge only reads and other calls that are safe to repeat.

Set the fan-out to 20 and the slow chance to 1%, then push the fan-out to 100. Turn on hedging and see how much of the tail disappears.

📌 Summary: Parts in series multiply availability down; replicas help only if they fail independently. Give every call a timeout and retry at one layer only. Circuit breakers, bulkheads and graceful degradation stop one failure spreading.

What Meera learns next

You cannot fix what you cannot see. Next: metrics, logs, traces and SLOs, so the next incident is caught by an alert in minutes, not by a support queue.

Check your understanding

Checkout needs a load balancer (99.99%), the checkout service (99.9%) and the orders database (99.9%), all in series. Roughly how available is checkout?
The team adds a second database copy in the same rack, on the same power feed, updated by the same config pipeline. Each copy is 99.9%. What is the honest availability of the pair?
The app, the API gateway and the checkout service each make up to 3 attempts when a call fails. During a database blip, how many database calls can one customer tap cause?
Recommendations start answering in 30 seconds instead of 50 ms, and checkout servers run out of threads. What is the best first fix?