Real-time updates: polling, long polling, SSE and WebSockets

Part of the free System Design course on LogicWiz, in the chapter: Networking and the entry layer. The interactive whiteboard exercises and simulations run in the app.

The map that kept asking

"The rider's dot jumped every five seconds, and nearly every request our servers answered said 'nothing new'."

Kabir, looking at the tracking service's traffic graph

Two screens that need news from the server

A customer in Pune orders milk and bread. Her app opens the tracking screen. A map shows the rider's dot, which should glide toward her door. Below it sits a support chat, in case the order is late. Neha, a SwiftKart support agent, may write to her at any moment.

Both screens need news the customer never asked for. The rider moved. Neha wrote. The order was packed.

Here is the difficulty. Ordinary web traffic uses HTTP, which works as request and response: the client asks, the server answers, and the exchange is over. The server can never start a conversation. It is like a shop counter: the shopkeeper serves whoever walks up, but cannot ring a customer at home.

So how does news travel from the server to the phone? There are four answers, from simplest to most capable. This lesson takes them one at a time.

Short polling: asking again and again

The simplest answer is short polling: the app asks on a timer. Every 5 seconds it sends "any news for order 123?" The server replies at once, usually with "no change".

sequenceDiagram
  participant A as App
  participant S as Server
  A->>S: any news? (0 s)
  S-->>A: no change
  A->>S: any news? (5 s)
  S-->>A: no change
  A->>S: any news? (10 s)
  S-->>A: order packed

It is easy to build and works with every server and network. Now count the waste with SwiftKart's numbers. An order is tracked for about 20 minutes. Its status changes about 8 times (4 million updates a day across 500,000 orders).

Polls per order = 20 min × 12 per min = 240; useful ≈ 8; wasted ≈ 232, about 97%

Across 500,000 orders, that is about 120 million requests a day to deliver 4 million real changes. Kabir also asks for a worst case. Suppose all 2 million daily users had the app open at once, say during a festival-sale countdown. Polling every 5 seconds gives 2,000,000 / 5 = 400,000 requests a second, and almost all of them say "no change".

Polling also forces a bad choice. Poll every 5 seconds and the rider's dot jumps, up to 5 seconds late. Poll every second for a smooth dot and the request count grows five times. You can have fresh or cheap, but not both.

Short polling is still the right tool for some screens. A page that changes rarely, like an admin report refreshed every 30 seconds, needs nothing cleverer.

Long polling: ask once, wait for news

Long polling changes one thing: the server does not answer at once. The app asks "any news?", and the server holds the request open until something happens, or until about 30 seconds pass. Then it answers, and the app asks again straight away.

Think of a child on a car trip. Short polling is "are we there yet?" every five minutes. Long polling is the parent saying "I will tell you when we arrive", and the child asking again only after each answer. The analogy stops at one point: the parent never gets tired, but a server pays a little memory for every request it holds open.

sequenceDiagram
  participant A as App
  participant S as Server
  A->>S: any news?
  Note over S: waits, nothing yet
  S-->>A: Neha wrote a message (after 12 s)
  A->>S: any news?
  Note over S: waits 30 s, nothing
  S-->>A: no change, ask again

News now arrives within moments, and empty answers drop to about two a minute (the 30-second timeouts). A quiet support chat works well this way. The cost: every waiting request holds an open connection on the server, and every message still needs a fresh request. If news comes every second, long polling turns back into ordinary polling.

Server push: server-sent events and WebSockets

The last two answers keep one connection open, so the server can send news the moment it has some.

Server-sent events (SSE) are one-way. The app makes one ordinary HTTP request, and the server never finishes the response. It keeps writing small events into it, one per change: "rider at 18.52, 73.85". If the connection breaks, the browser or the app's SSE library reconnects by itself and says which event it saw last, so nothing is missed. When the customer needs to send something, the app makes a normal HTTP request alongside.

That fits the tracking map. The customer's phone only listens. The rider's app sends a location ping every 3 seconds over plain HTTP, and the server forwards each one down the open stream. The dot moves within about a second, with no wasted requests.

WebSockets are two-way. The app sends one HTTP request asking to "upgrade" the connection. The server agrees, and from then on there is no request and response. It is an open pipe: either side can send a message at any moment.

sequenceDiagram
  participant A as App
  participant G as WebSocket gateway
  A->>G: HTTP request, please upgrade
  G-->>A: agreed, connection stays open
  G-->>A: Neha is typing
  A->>G: customer message
  G-->>A: Neha's reply

A WebSocket is like a phone call that stays connected: either person can speak whenever they like. Unlike a call, silence tells you nothing. A quiet connection may be idle, or it may have died.

The support chat suits a WebSocket. Both people type, and small signals like "Neha is typing" flow both ways many times a minute. A WebSocket gateway is a server whose job is to hold these open connections and pass messages between them and the services behind.

What an open connection costs a server

An open connection lives in the server's memory for as long as the screen is open. It needs a buffer (memory set aside for data in transit) and a little state about the user. Plan on tens of kilobytes each. A well-tuned server holds tens of thousands of connections; a fair planning figure is about 50,000.

Size SwiftKart's tracking screens. 500,000 orders a day, each watched for about 20 minutes, make 10 million screen-minutes. Spread over the 1,440 minutes in a day, that is about 7,000 screens open at any moment. The evening peak is roughly four times that, about 30,000.

30,000 connections × ~30 KB ≈ 900 MB; 30,000 / 50,000 per server ≈ 1 server, so run 3 for failures and deploys

A connection that lasts minutes instead of milliseconds brings three more costs.

  • Load balancing is sticky. The load balancer picks a gateway once, when the phone connects. Every later message on that connection goes to the same gateway. A gateway added at 7 p.m. gets only new connections, so load evens out slowly.
  • Dead connections look alive. A phone that enters a tunnel can vanish without saying goodbye. So both sides send a tiny heartbeat ("still here") every 30 seconds or so, and close the connection when heartbeats stop.
  • Deploys cause reconnect storms. Restart a gateway holding 30,000 connections, and 30,000 phones reconnect in the same second. Each one repeats the login check and asks for the latest state, so a calm system suddenly gets a flood.

The storm has a known cure. Phones reconnect with backoff and jitter from the rate-limiting lesson: wait longer after each failure, plus a random extra delay. Gateways also close their connections gradually over a few minutes, which is called draining.

⚠️ Warning: A phone app in the background usually loses its open connection soon. For news that must reach a closed app, such as "your order has arrived", send a push notification instead.

One puzzle remains. When the tracking service has a new location, it must find the gateway holding that customer's connection. The chat app chapter solves this with a session store.

Picking one: a simple decision guide

In a mock interview, a candidate said: "WebSockets are always better, so I use them everywhere." Kabir disagreed. A WebSocket is the most capable choice, and also the most expensive to run: memory per connection, sticky balancing, heartbeats and reconnect storms. "My orders", the delivery estimate and the product page change rarely. A plain request when the screen opens is cheaper and simpler. Choose the lightest tool that meets the need.

flowchart TD
  Q1{"News needed within seconds?"} -->|"no"| P["Short polling or a plain request"]
  Q1 -->|"yes"| Q2{"Does the client also send often?"}
  Q2 -->|"yes"| W["WebSocket"]
  Q2 -->|"no"| Q3{"News many times a minute?"}
  Q3 -->|"yes"| E["Server-sent events"]
  Q3 -->|"no"| L["Long polling"]

Run SwiftKart's screens through it:

  • Tracking map: news every few seconds, and the phone only listens. Server-sent events fit.
  • Support chat: both sides send. A WebSocket fits.
  • Order history: changes rarely. A plain request when the screen opens.

Many apps already hold one WebSocket for chat and reuse it for tracking. That is fine. The point is to have a reason you can say out loud.

💡 Tip: In an interview, give the connection count and the per-server figure. "About 30,000 open screens, about 50,000 per gateway, so three gateways for safety" shows you know what the choice costs.

Now build the live tracking path yourself.

📌 Summary: Polling asks repeatedly and wastes most requests. Long polling waits for news. SSE pushes one way; WebSockets carry both ways. Open connections cost memory, sticky balancing and reconnect storms.

What Meera learns next

News can now flow both ways through the front door. Next week: what the app and the services actually say to each other. Meera designs clean APIs and models the data behind them.

Check your understanding

  1. An order is tracked for about 20 minutes and its status changes about 8 times. The app polls every 5 seconds. Roughly what share of those polls bring nothing new?
  2. The tracking map only shows the rider's dot. The customer's phone never sends anything on that screen, and the dot changes every few seconds. Which fits best?
  3. SwiftKart restarts a WebSocket gateway that holds 30,000 connections during a deploy. Seconds later the login service and the database raise alarms. What is the best fix?
  4. Three gateways are full, so the team adds a fourth at 7 p.m. An hour later the new gateway still holds far fewer connections than the others. Why?