Rate limiting and the entry layer that fails

Part of the free System Design course on LogicWiz, in the chapter: Networking and the entry layer. The interactive whiteboard exercises and simulations run in the app.

Week 3, Day 4: Ten thousand taps a second from one phone

"A buggy app version retried every failed request instantly, forever. One user's phone took down the page for everyone."

Kabir, on the worst retry loop he has seen

Why we limit

Here is the failure. A SwiftKart app update has a bug: when a request fails, it retries immediately, with no pause. On a bad network, one phone sends hundreds of requests a second. Multiply by the thousand phones on that app version, and the service drowns in traffic that is not even useful.

A rate limiter says: each caller gets a fixed allowance of requests over time. Past it, requests are turned away cheaply, before they reach the expensive parts. It protects against:

  • bugs like this one;
  • scrapers (scripts that copy your data);
  • password guessing;
  • honest customers who, all together, send more traffic than you planned for.

The token bucket, with real numbers

The most common algorithm is the token bucket, and it is easy to picture.

Imagine each user has a bucket holding up to 10 tokens. Every request costs 1 token. The bucket refills at 2 tokens per second, never beyond 10.

Watch one user. The bucket starts full at 10. She opens the app and fires 10 requests in a burst: all allowed, bucket empty. The 11th request arrives immediately: no token, so it is rejected. One second later, 2 tokens have dripped in, so 2 more requests pass. Over a long time she can sustain 2 requests per second. After a quiet spell, she is allowed a burst of up to 10.

Two numbers tune it: the refill rate (the steady speed you allow) and the bucket size (how big a burst you forgive). Real apps send requests in bursts. Because the bucket forgives a burst, it suits them better than a rigid "no more than 2 in any second".

Set the bucket to 10 and the refill to 2 below. Then switch the traffic from steady to burst to spike. Watch in which seconds requests start getting rejected with a 429. Then shrink the bucket to 2 and see how the same burst is treated.

Compare a fixed window: "100 per minute, counter resets on the minute". It is simpler, but a caller can send 100 requests at 12:00:59 and 100 more at 12:01:00. That is 200 in two seconds. The bucket avoids this problem at the window edge.

flowchart LR
  R["Refill: 2 tokens/s"] --> B["Bucket, max 10"]
  Q["Request"] --> B
  B -->|"token left"| OK["Allow"]
  B -->|"empty"| NO["Reject 429"]

What a 429 means

When the limit is hit, the server answers with HTTP status 429 Too Many Requests. It is a deliberate "slow down" and not a crash (that would be a 5xx). Good limiters add a Retry-After header telling the client how many seconds to wait.

Well-behaved clients respond by backing off. They wait, retry, and wait twice as long if it fails again. They also add a little randomness, so a thousand clients do not all return at the same instant. This is called exponential backoff with jitter (jitter is the random part). It is exactly what SwiftKart's buggy version lacked.

💡 Tip: Rejecting is cheap, which is the point. A 429 costs a counter lookup; the request it replaced would have cost a database query. Put the limiter before the expensive work, not after.

Where to enforce it

You can limit at several places, and the choice is a design decision you should state.

  • At the gateway or edge. One place, in front of everything. Best for per-user and per-IP limits and for stopping abuse early.
  • Inside each service. Useful for a limit tied to that service's own cost, such as "5 password attempts per hour".
  • On the client. Politeness, not protection. Never trust it, because a malicious caller ignores it.

A limiter needs to count across all your gateway instances. If you have four gateways each counting separately, a user can get four times the allowance. So the counters usually live in a shared fast store such as a cache, and every instance asks it. Updates must be atomic: done as one step that cannot be split, such as a single increment. Otherwise two gateways can both read "one token left" and both allow the request. Decide what to do if that store is down. Allowing requests (fail open) keeps sales flowing. Blocking them (fail closed) protects the backend. Pick per endpoint and say why.

Most floods come from friends

Meera assumed limits exist to stop hackers. Kabir's list of real floods had few attackers in it:

  • a retry bug in the SwiftKart app;
  • a partner's script polling every second;
  • a festival promotion that made honest users refresh again and again. Limits are fairness and safety for everyone. Without them, the loudest caller gets all the capacity and the quiet ones get errors.

The entry layer is a single point of failure

Back to Meera's diagram. Everything passes through the limiter, the gateway and the balancer. If any of them is one machine, one crash takes the entire product down. It does not matter how many service copies stand behind it.

The fix is the same idea as for databases: no component without a spare. Run the balancer as a pair, so one can take over if the other fails. Run the gateway and limiter as several instances behind it. Use more than one DNS provider, or a managed service with its own redundancy. Then ask the hard question out loud: "What happens when the entry layer itself fails?" Interviewers value a candidate who raises it without being asked.

⚠️ Warning: Redundancy has a price: shared counters, session handling and deployment get harder. Mention it as a trade-off, not a free win.

Try it: protect the service from the retry loop, with the counters in a shared store.

📌 Summary: A token bucket allows a steady rate plus short bursts. Reject excess with a 429, and keep counters in a shared store. The limiter, gateway and balancer each need a spare.

What Meera learns next

Traffic now arrives safely at the right service. What does a caller say to it? Next week: how to design APIs, paginate lists, avoid double payments, and model the data behind them.

Check your understanding

  1. A user's bucket holds 10 tokens, refills at 2 per second and is full. Her app fires 15 requests in the same instant. What happens?
  2. With the same bucket (10 tokens, refill 2 per second), a script hammers the API nonstop for ten minutes. About how many requests a second get through over that time?
  3. SwiftKart runs four gateway instances, and each keeps its own token counters in local memory. What goes wrong?
  4. SwiftKart's app gets a 429 with "Retry-After: 3". How should a well-behaved client respond?