From one server to a million users

Part of the free System Design course on LogicWiz, in the chapter: System design from zero. The interactive whiteboard exercises and simulations run in the app.

One machine was enough, until it was not

"Our whole company ran on one computer. Then a cricket match ended, everyone ordered dinner, and it stopped answering."

Kabir, on SwiftKart's first year

Day one: one machine does everything

Kabir tells Meera how SwiftKart began. It launched in two Bengaluru neighbourhoods with about 5,000 customers. One rented computer in a data centre ran everything: the server program and the database, side by side. At dinner time it received about 5 requests a second. Most of the day it sat idle.

That was the right choice. One machine is cheap, easy to understand and quick to change. Kabir's rule: start with the smallest thing that works, and add a part only when a number forces it.

This lesson is the map of the whole course. Each step below is a problem SwiftKart hit as it grew, in the order it hit them. For each one you will see what broke, what was added, and what the new box does. Each box gets its own chapter later, so here you only need the idea.

Step 1: give the database its own machine

At about 50,000 customers, a new release had a bug: the server program slowly used up all the machine's memory. The machine froze, and the database on it froze too. For 40 minutes nobody could order. Even on good days, the two programs competed for the same processor and memory, and evening pages slowed from about 100 ms to over a second.

The fix was to move the database to a second machine. Now a crash in the server program cannot take the data down with it. Each machine can be sized for its own job, and the server program can be restarted or updated while the database keeps running.

Step 2: a bigger machine, then more machines

At about 300,000 customers, the server machine ran at 90 percent of its processor every evening. There are two ways to add capacity.

Vertical scaling means making one machine bigger: more processor cores (the parts of the chip that run code) and more memory. SwiftKart moved from an 8-core machine to a 32-core one. No code changed, and evenings were smooth again. In a restaurant, this is hiring a faster cook.

Vertical scaling has a ceiling. The biggest machine you can rent has a fixed size. Price climbs faster than size, so the last doubling costs the most. And it is still one machine: if it fails, or you restart it to upgrade, everything stops.

Horizontal scaling means adding more machines of the same kind, each running a copy of the server program. In the restaurant, this is opening more counters. At about 1 million customers, the evening peak reached about 3,000 requests a second. One copy handled about 1,000. So SwiftKart ran four copies, one of them spare.

Now the phones need to know which copy to call. A load balancer sits in front of the copies. The phones see one address, and the load balancer passes each request to a healthy copy. The entry-layer chapter covers how it chooses.

Horizontal scaling has one rule: the copies must be stateless. A stateless server keeps nothing about a customer in its own memory between requests. SwiftKart learned this the hard way. The first version kept each cart in server memory. A customer added milk on copy 1, the next tap reached copy 3, and the cart looked empty. The fix was to keep carts in a shared store that every copy reads. Then any copy can answer any request, and a dead copy loses nothing.

Step 3: replicas and a cache for the reads

Today SwiftKart has 2 million daily users. Order status is read about 20 million times a day and updated about 4 million times, so about 5 reads for every write. Reads peak near 1,000 a second on a normal evening, and near 2,000 on the night of the IPL final. The main database is comfortable at about 2,000 simple queries a second. Add the catalogue reads and the logins, and it ran close to its limit on every match night. Worse, it was still one machine. One failed disk would stop every order.

A replica is a second database that keeps a continuous copy of the main one, which is called the primary. All writes go to the primary, and the replicas copy each write a moment later. Reads can go to the replicas, which spreads the load. If the primary dies, a replica is promoted to take its place. That switch is called failover. The databases chapter covers replicas in depth.

Many reads were also the same question asked again and again. The home page shows the same 200 popular products to everyone in a city. A cache is a faster copy of data you already have, kept in memory. The server looks in the cache first and asks the database only when the answer is missing. If 95 of every 100 home-page reads are answered by the cache, the database sees one twentieth of that traffic. The caching chapter covers when a cache helps, and when it serves old data.

Step 4: a CDN for files and a queue for slow work

Each home page loads about 30 product photos of about 50 KB each, roughly 1.5 MB in all. At 2,000 page opens a second during a sale, that is about 3 GB of photos a second leaving SwiftKart's servers. The network bill doubled, and photos loaded slowly for customers far from Mumbai.

A CDN (content delivery network) is a set of servers in many cities that keep copies of files such as images and app code. A phone in Guwahati gets its photos from a server nearby instead of from Mumbai. SwiftKart's own servers stop sending photos at all. The entry-layer chapter covers CDNs.

The next problem was checkout. After payment, the order service did five things before it answered: save the order, tell the store, send an SMS, make a PDF invoice and add loyalty points. The PDF alone took about 2 seconds. Customers stared at a spinner for 3 seconds, and some tapped Pay a second time.

A queue is a waiting line for work. Now the order service saves the order, puts a message "order 123 placed" on the queue, and answers in about 100 ms. Separate programs called workers take messages from the queue and do the slow jobs a few seconds later. If the SMS provider is down for a minute, its messages wait in the queue instead of failing the order. The queues chapter covers this.

Step 5: sharding, when writes outgrow one database

Replicas help reads, but every write still lands on the primary. Then SwiftKart began showing riders live on the map. Each rider's phone sends its location every few seconds: about 10,000 writes a second at the evening peak. The primary is comfortable at about 2,000. That is five times too many, and adding replicas does not change it.

Sharding means splitting the data across several databases, each holding one part, called a shard. Rider locations could be split by city: Mumbai's riders in one shard, Delhi's in another. Each shard takes only its share of the writes and has its own primary and replicas. The cost is real. A question that needs every shard, such as "all late orders in India today", becomes harder to answer. That is why sharding comes last. The databases chapter covers how to choose the split.

Here is the picture SwiftKart ended with.

flowchart LR
  P["Phones"] --> D["DNS"]
  P --> C["CDN: photos, app files"]
  P --> L["Load balancer"]
  L --> S["4 server copies"]
  S --> K["Cache"]
  S --> A["Shard A: primary + replica"]
  S --> B["Shard B: primary + replica"]
  S --> Q["Queue"]
  Q --> W["Workers: SMS, invoice"]

Meera's first reaction was, "So we should build this whole picture on day one." Kabir disagreed. Each box solved a problem SwiftKart had, at a number it had reached. Built for 5,000 customers, the same picture would have cost months of work and many idle machines. Every extra box is also one more thing that can break. The estimation chapter shows how to tell, from numbers, which boxes you need.

📌 Summary: Start with one machine and add a part only when a number forces it. More copies need a load balancer and stateless servers. Replicas and caches help reads; only sharding splits writes.

What Meera learns next

The story used words like latency, availability and stale data without pinning them down. Next, Meera learns the words every design uses, each with a number attached.

Check your understanding

1. SwiftKart now runs 4 copies of its server behind a load balancer. Customers add milk to the cart, tap again, and see an empty cart. What is the most likely cause?
2. The server already runs on the biggest 32-core machine the team can rent, and evening traffic keeps growing. What is the next step?
3. Rider location pings reach about 10,000 writes a second, and the primary is comfortable at about 2,000. A teammate suggests adding four replicas. Why does that not help?
4. After payment, checkout makes a PDF invoice and sends an SMS before it answers, so customers wait 3 seconds. What should SwiftKart add?