The words every design uses

Part of the free System Design course on LogicWiz, in the chapter: System design from zero. The interactive whiteboard exercises and simulations run in the app.

Fast on average, slow for thousands

"The dashboard said the average was 109 milliseconds. Support said customers were waiting three seconds. Both were right."

Meera, after her first week on call

Latency and throughput: how long, and how many

Two numbers describe how fast a system is. Latency is how long one request takes, from asking to answer. In the first lesson, a Track order tap took about 60 ms. Throughput is how many requests the system finishes each second. SwiftKart's order service handles about 1,000 requests a second at the evening peak.

Picture the road from Andheri to Bandra in Mumbai. Latency is how long one car takes to drive it, say 25 minutes. Throughput is how many cars reach Bandra each hour, say 3,000. Widen the road from three lanes to five, and about 5,000 cars an hour get through. But each car still takes about 25 minutes. More lanes do not make one car faster.

Servers behave the same way. Adding server copies raises throughput, but it does not make a single request faster. To cut latency, you remove steps or shorten them. The analogy stops at one point. A car must drive the whole road every time. A request can sometimes skip the trip, when a nearby cache already has the answer.

Latency and throughput meet when a system is nearly full. On a jammed road every car slows down. On a busy server, requests wait in line and latency climbs fast. That gives two different problems:

A performance problem: the system is slow even for one user. A request takes 2 seconds on an idle server. Find and fix the slow step.
A scalability problem: one user is fast, but many users are slow. A request takes 50 ms at 100 a second and 2 seconds at 3,000 a second. Add capacity, or fix the part that runs out first.

Kabir's quick test: "Is it slow at 3 a.m. too?" If yes, it is performance. If it is slow only at dinner time, it is scalability.

Why the average hides the pain: percentiles

Meera's dashboard showed the tracking page's average latency: 109 ms. That looked fine. Then she looked at 100 real requests from one evening. 98 of them took 50 ms. Two took 3,000 ms, because they waited behind a slow database query.

The average is (98 × 50 + 2 × 3,000) ÷ 100 = 109 ms. No real request took 109 ms. The number describes nobody.

A percentile tells you more. Sort the requests from fastest to slowest. The p50, also called the median, is the time that half of the requests beat: here 50 ms. The p99 is the time that 99 of every 100 requests finish within: here 3,000 ms. Think of a bank where the average wait is 5 minutes. That sounds fine, until you learn that 1 customer in 100 waits an hour.

Meera's first thought was, "1 in 100 is rare, so it does not matter." Kabir did the arithmetic. SwiftKart serves about 20 million status reads a day. 1 percent of that is 200,000 slow page loads, every day. Those are the customers who phone support. That is why good requirements say "p99 under 300 ms", not "fast on average".

The tail gets worse when one page calls many services. The visualiser below shows it. Choose a 1% (p99) slow chance and a fan-out of 1, meaning the page calls one service: 1 request in 100 is slow. Then drag the fan-out to 20, about what the SwiftKart home page calls, and watch the share of slow pages climb.

Availability, nines and single points of failure

Availability is the share of time the system does its job, written as a percentage. Engineers talk in "nines". The percentages look close together until you turn them into minutes. A year has about 525,600 minutes.

99%: down 1% of the year, about 3.65 days.
99.9% ("three nines"): about 8.8 hours a year, or 43 minutes a month.
99.99% ("four nines"): about 53 minutes a year, or about 4 minutes a month.

Each extra nine cuts the allowed downtime by ten. Four minutes a month is too short for a person to wake up and fix anything. At four nines, machines must notice the failure and recover on their own.

Pick 99.9% and then 99.99% below, and read the downtime per month. Then put three parts in a chain and see the whole path fall below its weakest part.

A single point of failure is one part whose failure stops the whole system. Think of an island joined to the city by one bridge. Close the bridge, and nobody crosses. SwiftKart's lone database was one, and so was its lone load balancer. The fix is a second copy that can take over, on a different machine. The analogy stops here: a second bridge is expensive and slow to build, while a second server copy can be ready in minutes.

Reliability is different from availability. Availability asks whether it answers. Reliability asks whether it does the right thing every time, even when parts fail. A checkout that answers every request but charges 1 customer in 500 twice is available, but not reliable.

Durability and consistency: is the data safe, and is it fresh?

A customer pays ₹640 and sees "Order placed". One second later, the database machine's disk fails. Is the order gone?

Durability means that once the system says "saved", the data stays saved, even if a machine dies right after. A durable database has already written the order to a second disk on another machine, so nothing is lost. Think of a shop that writes each sale in its register in pen and keeps a photocopy in another building. Durability is not the same as availability. During a one-hour outage, the system is not available. If no order is lost, it is still durable.

Now a different surprise. A rider taps "Picked up" at 7:02:00. The primary database saves it at once. A replica, the copy that serves reads, receives the change about one second later. The customer's app reads from the replica at half a second past 7:02 and still shows "Packing". That old answer is a stale read.

Consistency, at this stage, means: does every reader see the latest write? Many people assume that once data is saved, everyone sees it at once. With copies, that is not automatic. Think of watching a cricket match on TV with a few seconds of delay, while a friend at the stadium texts "Six!" before you see it. The analogy has a limit: a TV delay is small and the same for everyone, while a replica's delay varies and, under heavy load, can grow to minutes. The consistency chapter shows how to choose how fresh each read must be.

The numbers every engineer should know

You do not need exact figures, only the rough size of each step:

Reading from memory (RAM): about 100 nanoseconds. A nanosecond is a billionth of a second.
Reading from an SSD, a fast disk with no moving parts: about 0.1 ms.
A round trip between two machines in the same data centre: about 0.5 ms.
A round trip from Mumbai to Delhi: about 30 ms.
A round trip from Mumbai to the US: about 200 ms.

Stretch them to human time. If a memory read took 1 second, an SSD read would take about 17 minutes. A trip inside the data centre would take over an hour. A trip to the US would take about 23 days.

Three habits follow. Keep hot data in memory, which is why caches exist. Keep servers close to users and data close to servers. And never call another continent once per item in a loop: 50 items at 200 ms each is 10 seconds of waiting.

Find each of these steps on the ladder below. Then turn on human scale and compare a memory read with a trip to the US.

Now use the words on a design. The board below has a single point of failure at every layer.

📌 Summary: Latency is how long one request takes; throughput is how many finish per second. Judge speed by p99, not the average. Turn nines into minutes, and remove every single point of failure.

What Meera learns next

Meera now has the words and the map. Next, she walks into her first design review and learns how a design interview runs, from the first question to the last box.

Check your understanding

The tracking page's average latency is 110 ms, yet support gets daily calls about a page that takes seconds. Which number should Meera look at first?
A report page takes 2 seconds even at 3 a.m., when almost nobody is using SwiftKart. What kind of problem is this?
SwiftKart promises 99.9% availability for the tracking page. Roughly how much downtime does that allow each month?
A rider taps "Picked up". For about a second, the customer's app still shows "Packing". The order was not lost. What happened?
A new feature checks 50 products, one at a time, against a service in the US, about 200 ms per round trip. Roughly how long does a customer wait?