DNS, CDN, load balancers and the gateway

Part of the free System Design course on LogicWiz, in the chapter: Networking and the entry layer. The interactive whiteboard exercises and simulations run in the app.

Week 3: The request before the request

"Our servers were fine. Our one load balancer was not, and nobody had drawn it on the diagram."

A SwiftKart SRE (site reliability engineer), in a post-mortem: the written review held after an outage

Follow one tap from the phone to the server

A customer in Pune taps "Track order". Before any SwiftKart code runs, a surprising amount happens. Walk through it in order. Each step is a box you may draw.

  1. DNS. The phone knows the name api.swiftkart.example, but networks route by number. DNS (the Domain Name System) is the phone book that turns the name into an address. It can also hand out different addresses to different callers, for example the address of the nearest region.
  2. CDN. The product images and the app's static files do not need to come from your servers at all. A CDN (content delivery network) keeps copies of such files in many cities. The Pune phone fetches a picture from a machine nearby, instead of one on another continent. A CDN is a cache placed close to users.
  3. Load balancer. Dynamic requests, such as "give me this order's status", go to your servers. There are many of them, so something must choose one. That is the load balancer.
  4. API gateway. This often sits right behind the load balancer. It is the one door that checks who you are, applies limits and sends the call to the correct internal service.
flowchart LR
  P["Phone"] --> D["DNS"]
  P --> C["CDN"]
  P --> L["Load balancer"]
  L --> G["API gateway"]
  G --> S["Order service"]

Load balancers, in plain words

Last week Meera worked out that the sale would need around 100 server instances. A client cannot know about 100 addresses. The load balancer is a single address in front of them. It spreads requests across the instances and stops sending to the broken ones.

Two decisions define a load balancer.

Layer 4 versus layer 7. The layers are levels of the network model. An L4 balancer works at the level of connections. It sees addresses and ports, not what the request says, so it is very fast and simple. An L7 balancer reads the request itself: the path, headers and cookies. So it can send /images to one group and /orders to another.

Both kinds can usually terminate HTTPS, meaning they decrypt it once at the door. The difference is what they can see afterwards. An L7 balancer can also pin a user to one server, called a sticky session. That is a workaround; prefer stateless servers, which keep no user data in their own memory. L7 costs a little more work per request and gives you much more control. A fair default: L7 for an HTTP API, and L4 when you want raw speed, for example for a database proxy or a game connection.

How to pick the next server. Round robin goes down the list in turn: server 1, 2, 3, 1, 2, 3. It is perfect when requests cost about the same. Least connections sends each new request to the server with the fewest active connections right now. This is better when some requests are slow. Imagine one instance stuck on an expensive report. Round robin keeps feeding it its share. Least connections sends new requests to the other instances instead.

Health checks: how it knows a server is dead

A balancer that sends traffic to a crashed server is worse than none. So it runs health checks: every few seconds it calls something like GET /health on each instance. If three checks in a row fail, it removes that instance from rotation and stops sending requests there. When checks pass again, it puts the instance back.

The check should test something real but cheap. A health endpoint that always replies "ok" will pass even when the database connection is broken. The balancer will then keep sending users to an instance that only returns errors.

💡 Tip: Distinguish "the process is up" from "this instance can serve". Do not fail the check only because a shared dependency, such as the database, is down. Then every instance leaves rotation at once, and the whole fleet disappears. Check the instance itself, cheaply.

The API gateway: one door for many services

If SwiftKart has separate orders, payments and catalogue services, you do not want the phone app to know three addresses and log in three times. The API gateway gives clients one entry point. It handles authentication (who are you?), request limits, logging and routing: /orders/* goes to the orders service, /pay/* goes to payments.

It overlaps with a Layer 7 load balancer, which is why interviewers often accept either box. The usual split: the load balancer spreads traffic across copies of one service; the gateway chooses which service a request belongs to and applies policy. In small designs, one component can do both jobs. Say so out loud, rather than drawing two boxes you cannot justify.

A load balancer does not add capacity

Kabir hears this sentence at least once a month: "We put a load balancer in front, so it scales." The balancer only spreads load across servers that already exist. If all 100 instances share one database, the database is still the ceiling from last week. And if the balancer itself is one machine, it becomes a new single point of failure: one part whose failure stops everything.

Scale comes from the whole path: cache the repeatable reads, balance the stateless servers, and make sure the data layer can follow. The balancer is one link in that chain.

⚠️ Warning: Servers behind a balancer should be stateless: no user session living only in one server's memory. Otherwise the next request lands on a different server, and the user appears "logged out". Keep sessions in a shared store, or in a signed token the client carries.

Now build the front door yourself.

📌 Summary: A load balancer spreads requests over stateless servers and uses health checks to skip dead ones. Use least connections when some requests are slow. A balancer spreads load; it does not create capacity.

What Meera learns next

The door is built and spreads traffic. But what if one app, or one angry script, sends ten thousand requests a second through it? Next: rate limiting, and how to keep the front door itself from failing.

Check your understanding

  1. Most SwiftKart requests take about 50 ms, but a few report requests take 5 seconds and tie up whichever instance gets them. Which way should the balancer pick the next server?
  2. Each instance's /health check runs a query on the shared orders database. The database is down for 30 seconds. What happens?
  3. A team says "we added a load balancer, so we scale". All 100 instances behind it share one database that is already at its ceiling. Which statement is true?
  4. After SwiftKart puts three instances behind a balancer, users log in and are told they are logged out on their very next tap. What is the most likely cause?