Why and where to cache

Part of the free System Design course on LogicWiz, in the chapter: Caching. The interactive whiteboard exercises and simulations run in the app.

Week 6: The product page that asked the same question a million times

"The catalogue never changes in the middle of the day, yet we ask the database for the same milk price 3,000 times a second."

Kabir, reading the dashboard

A cache is a faster copy of data you already have

Open SwiftKart's product page for "Amul toned milk 500 ml". The name, price and photo are the same for every shopper for hours. On a normal evening the page is requested about 2,000 times a second. Each request runs a database query that takes about 8 milliseconds. The query joins a few tables, so the database can comfortably run only about 500 of them a second. Simple single-row lookups would go several times faster.

A cache is a small, fast store (usually memory) that holds copies of answers you have already worked out. The second person to ask for the milk page gets the copy, in about 1 millisecond, and the database never hears about it.

The most common pattern is cache-aside (also called lazy loading). In plain words, the service does this:

  1. Ask the cache for the milk page.
  2. Hit: it is there, return it. Done.
  3. Miss: it is not. Ask the database, put the answer in the cache for next time, return it.
flowchart LR
  S["Service"] -->|"1 ask"| C["Cache"]
  S -->|"2 on miss"| D["Database"]
  S -.->|"3 store copy"| C

The application owns the logic. The cache only stores and returns entries; the service decides what goes in it and when.

Hit ratio: the one number that matters

The hit ratio is the share of requests the cache answers. Real arithmetic, using the milk-page traffic of 2,000 requests per second:

  • Hit ratio 0%: the database sees 2,000 queries a second.
  • Hit ratio 90%: the database sees 200 a second.
  • Hit ratio 99%: the database sees 20 a second.

Going from 90% to 99% looks like a small gain, but it cuts database load by another tenfold. The database is shielded from the miss rate, not the hit rate, so small changes near 100% matter enormously. Say the database safely handles about 500 of these queries a second. At 2,000 requests per second, a 90% hit ratio is comfortable. At the festival-sale peak of 10,000 per second, the database would see 1,000 and fall over. At 99% it sees 100, well within its limit.

Latency improves too. Average time at 90% hits is 0.9 x 1 ms + 0.1 x 9 ms (a miss pays both trips), roughly 1.8 ms, versus 8 ms with no cache.

Try it below. Set requests to 2,000 a second, then drag the hit ratio from 90% to 99% and watch database queries a second fall tenfold. Then raise traffic to the 10,000 sale peak and find the hit ratio that keeps the database under 500.

Meera also noticed the cache saves money, not only time. Most reads go to a small hot set: the few thousand products people actually look at. The hot set fits in a small amount of memory. Memory costs more per gigabyte than disk, but you need very little of it. So one small cache node can take most of the read load off a much bigger, costlier database. Kabir calls this the cheapest way to scale reads, as long as the hit ratio is high.

TTL: how long a copy may live

How long is a copy trusted? Each entry gets a TTL (time to live), a countdown after which it is thrown away and the next request refetches. Milk prices change rarely, so a 10-minute TTL is fine. Pair it with a delete-on-write when marketing changes a price (next lesson). At worst, a shopper sees a price 10 minutes old. A rider's live location needs a TTL of a couple of seconds. Pick the TTL from the question Kabir taught in Week 1: how stale can this be before someone is harmed?

💡 Tip: Short TTL means fresher data but a lower hit ratio. Long TTL means a higher hit ratio but older data. The business decides the staleness; you do the arithmetic.

Where caches live

A request passes through several places that can each hold a copy. From closest to the user:

  • Client: the app or browser keeps images and recent responses. Free and instant, but you cannot reach in and clear it.
  • CDN (content delivery network): servers near users in Indian cities serve product photos without touching your data centre. Great for static files.
  • Application cache: a map inside the service's own memory. Fastest, but each of 20 service machines has its own copy, so they can disagree.
  • Shared cache (Redis or similar): one in-memory store all service machines use. Slightly slower than local memory (a network hop of about 1 ms) but everyone sees the same copy.

A good design uses several: CDN for photos, shared cache for product data.

Write-through and write-back

Reads are the easy half. When data changes, what happens to the copy?

  • Write-through: every write goes to the cache and the database before it is acknowledged. The cache stays fresh for writes that go through this path, and once the database commit succeeds, a cache crash loses nothing. Cost: each write pays for both.
  • Write-back (write-behind): the write goes to the cache, which says "done" at once and saves to the database a moment later in a batch. Writes are very fast, but if the cache dies before saving, those writes are lost forever. This suits low-value data such as view counters, never payments.

A candidate in a mock interview said "A cache makes everything faster, so cache everything." Caching data that is rarely re-read wastes memory and gets almost no hits. A cache pays off for data read many times per write and shared by many users. A user's unique payment history, read once, is a poor candidate.

📌 Summary: The database sees only the misses, so 99% hits means a tenth of the load of 90%. TTL trades freshness for hits. Write-through is safe; write-back is fast but can lose data.

What Meera learns next

A cache is easy to add and surprisingly easy to get wrong. Next: stale data, stampedes on sale day, hot keys, and why the cache must never be the only copy.

Check your understanding

  1. SwiftKart's product pages get 5,000 reads a second through a cache-aside cache. The hit ratio rises from 95% to 99%. What happens to queries reaching the database?
  2. Meera wants to cache each customer's full payment history page. Each customer opens it about once a month. Is it worth caching?
  3. A teammate proposes write-back caching for wallet top-ups so the API can reply faster. What is the problem?
  4. Which TTL fits a rider's live location shown on the order tracking screen?