Seeing your system

Part of the free System Design course on LogicWiz, in the chapter: Reliability and observability. The interactive whiteboard exercises and simulations run in the app.

Week 9, Day 4: The alert that never fired

"The dashboard was green for forty minutes while UPI checkouts failed. CPU was fine. Customers were not."

Rohit, SwiftKart's on-call SRE

Metrics, logs and traces, with one incident

Wednesday, 7:50 pm. A change to the payment service's retry settings went out. From 7:52, about one checkout in eight failed. Nobody noticed until support tickets piled up at 8:30. Rohit was the SRE (site reliability engineer) on call that night. He walked Meera through how they found the problem, using three different tools.

  • Metrics are numbers over time: checkout requests per second, errors per second, p99 latency (the time within which 99 of 100 requests finish). They are cheap to keep for everything. They told Rohit that something was wrong: the checkout error rate jumped from about 0.2% to about 12% at 7:52.
  • Logs are one line of text per event, with details: "order 81723: UPI payment failed: gave up waiting for bank". They told him what was failing: one kind of error, only on UPI.
  • Traces follow one request across every service it touches, with the time spent in each hop. A trace of a failed checkout showed the payment service giving up on the bank after 400 ms. It retried twice and failed. But about one bank answer in eight normally takes up to a second. That told him where and why.

In plain words: metrics say something is wrong, logs say what happened, traces say where in the chain it happened. Metrics stay cheap at any scale. Logs and traces for 2 million users are expensive. So teams keep a sample of normal requests and try to keep all the failed ones.

The four golden signals

What should you measure? For every service users touch, start with four numbers, called the four golden signals:

  1. Latency: how long requests take. Use percentiles: p50 for the typical user, p99 for the unlucky one. Track failures separately, because a fast error is not a fast success.
  2. Traffic: how much demand there is: requests per second, orders per minute.
  3. Errors: the share of requests that fail, including "succeeded but wrong" ones, such as a 200 OK with an empty cart.
  4. Saturation: how full the system is: thread pools, connection pools, queue lengths, disk. It warns you before latency does.

On Wednesday, latency, traffic and saturation all looked normal. Only the error signal showed the problem. That is why you watch all four.

SLIs, SLOs and the error budget

Three related words, from the user's point of view:

  • An SLI (service level indicator) is a measurement of what users feel: "the share of checkout requests that succeed", "the share answered within 800 ms".
  • An SLO (service level objective) is the target for that SLI over a window: "99.9% of checkouts succeed, measured over 30 days".
  • The error budget is the part you are allowed to fail: here, 0.1%.

The arithmetic. A 30-day month has 30 × 24 × 60 = 43,200 minutes, and 0.1% of that is about 43 minutes of full outage. Counted in requests: about 500,000 orders a day is roughly 15 million checkouts a month. So the budget is about 15,000 failed checkouts.

Now Wednesday. At an evening rate of about 20 checkouts a second, 40 minutes is 20 × 2,400 = 48,000 checkouts. At 12% failing, that is about 5,800 failures: nearly 40% of the month's budget, gone in one evening.

The budget turns an argument into a rule. While budget remains, ship features. When it is nearly spent, slow risky releases and spend the time on reliability. Product and engineering agreed the number in advance, so nobody has to argue about it during an incident.

In the first SLO meeting, a product manager asked: "Why not aim for 100%?" Because it cannot be reached. Phones lose signal and banks have bad minutes, so some failures are outside your control. Chasing 100% would also freeze every change. Pick the lowest target users will not notice: perhaps 99.9% for checkout and 99% for recommendations.

Alert on what users feel

Why did no alert fire? SwiftKart's alerts watched causes: CPU above 80%, memory above 90%, a server down. This failure caused none of them. The rule: page a human (send an urgent alert to their phone) on symptoms users feel. Those are the error rate, latency against the SLO, or the error budget burning too fast. "Page if more than 2% of checkouts fail for 5 minutes" would have woken Rohit at about 7:57, not 8:30. Causes such as CPU and disk belong on dashboards and in low-urgency tickets.

Every page should be urgent, real and actionable. Picture an alert that fires daily and needs nothing done. It teaches people to ignore alerts, including the real one.

💡 Tip: Put the golden signals for each service, drawn against its SLO, in the top row of its dashboard. If you have to scroll to learn whether users are hurting, the dashboard has the wrong order.

On call and the blameless post-mortem

Being on call means owning the pager for a week. It works when each alert has a runbook: what the alert means, the first three things to check, and how to roll back. The first job in an incident is to stop the damage, here by rolling back the 7:50 change. Finding the root cause comes later.

Afterwards the team writes a post-mortem, a written review of the incident. It has:

  • a timeline;
  • the impact in numbers (about 5,800 failed checkouts);
  • what went well and what went badly;
  • actions, each with an owner.

It is blameless: it asks what about the system let a mistake through, not who made it. A careful engineer shipped a timeout shorter than the bank's slower, perfectly normal answers. The system let it ship with no symptom alert and no canary release (a release sent first to a small share of servers and watched). The actions: an alert on the checkout error rate, canary releases checked against the SLO, and a check that flags any timeout shorter than the callee's p99. Blame teaches people to hide mistakes. Blameless reviews teach them to report mistakes early.

📌 Summary: Metrics show that something is wrong, logs show what, traces show where. An SLO targets what users feel; 99.9% leaves an error budget of about 43 minutes a month. Page on symptoms, not causes.

What Meera learns next

SwiftKart can now survive and see failures. Next week, she opens the database itself: why it reads orders so fast, and why rider location pings overload its disks.

Check your understanding

  1. Checkout has an SLO of 99.9% success over a 30-day month. Roughly how much full outage does the error budget allow each month?
  2. Which of these should page the on-call engineer at 3 am?
  3. Metrics show checkout p99 latency doubled, but only for some users. Which tool best shows which service in the chain is adding the time?
  4. With 10 days left in the month, checkout has used 90% of its error budget. What does the error-budget policy suggest?