Video streaming: deep dives, failures and follow-ups

Part of the free System Design course on LogicWiz, in the chapter: Design video streaming. The interactive whiteboard exercises and simulations run in the app.

Week 15: Where the video pipeline hurts

"The new series went live at 8 p.m. A million people pressed play in ten minutes. Our servers were fine. The storage behind the CDN was not."

Kabir, in a launch-night review

Deep dive 1: the transcoding pipeline

Weak: one worker takes a whole video and encodes it in order. A crash at minute 40 restarts from zero. A 2-hour film takes about 10 hours for 5 renditions, and it blocks the worker for everyone behind it.

Good: split the video into 6-second chunks and queue one job per chunk and rendition. A 10-minute video is 500 jobs. Workers run them in parallel, and a failed job is retried on its own. Each job writes to a fixed key, so running twice is harmless. A job that fails 5 times goes to a dead-letter queue, a side line for people to inspect. A small status table counts finished jobs per video.

Great: add priority. Use a fast lane for the lowest rendition, so the video is playable early, and a slower lane for 1080p. Scale workers by queue depth and the age of the oldest job, not by a fixed number. The 21,000 cores of lesson 1 are an average. Evening uploads may double it, so paying for 42,000 cores all day wastes money. Before marking a video ready, check each rendition: the duration matches the original and every segment exists.

Trade-off: smaller chunks mean more parallelism but more overhead per job, and each chunk starts with a costly keyframe, so files are a little larger. Segments of 2 to 10 seconds are common, and 6 seconds sits in the middle.

Deep dive 2: CDN cost and the long tail

Delivery is 70 PB a day, so it is the main cost and the main speed lever. Assume the top 1% of videos get about 80% of views. This is an assumption, but real catalogues are steep. The rest is the long tail: millions of videos each watched rarely.

Weak: serve segments from the origin store. 15 Tbps would overwhelm it, and every viewer's bytes would leave from one place, far from most of them.

Good: put a CDN in front. Popular segments stay at the edge. Say the edge answers 90% of requests. Then 10% of 15 Tbps = 1.5 Tbps still reaches the origin. The long tail is the problem, because a title watched once an hour is rarely in cache.

Great: add layers. Edge servers send misses to a regional tier, a bigger shared cache. Say it catches 80% of those misses. Then the origin sees 15 x 0.1 x 0.2 = 0.3 Tbps. Use request collapsing: when 1,000 viewers want the same new segment, the cache makes one request to the layer behind it and shares the answer. Before a big release, pre-warm: load the first segments of the new title into the edges, using a CDN feature or by requesting them through each region. Then the first viewers do not all cause a cache fill, the traffic that populates a cache. Disk is limited, so drop the least recently used files first.

Where to put edges? In the big cities, and, for the heaviest traffic, inside large internet providers' networks, so video does not cross the wide internet. Trade-off: your own edge servers cost money to run and plan. A service starts with a CDN provider and adds its own when volume pays for it.

Deep dive 3: start-up time

Budget: under 2 seconds. The playback call takes about 100 ms and the manifest about 100 ms. The rest is the first segment. These sums ignore connection setup, which adds a little.

Weak: start at the highest quality. A 6 Mbps segment of 6 seconds is 4.5 MB. On a 4 Mbps link that takes 4.5 x 8 / 4 = 9 seconds. The viewer stares at a spinner.

Good: start low and climb. At 0.8 Mbps the first segment is 0.6 MB. On the same link it downloads in 0.6 x 8 / 4 = 1.2 seconds. Total: 0.1 + 0.1 + 1.2 = about 1.4 seconds. The next segments use the measured speed.

Great: also cut the first segments shorter, for example 2 seconds, so playback starts after less data. Remember the last measured speed on this device, so the first pick is not blind. Fetch the manifest early, when the viewer opens a title page. Use a rule that stops the quality flipping up and down at every segment.

Trade-off: the first seconds look blurry, and shorter segments mean more requests. Viewers accept a blurry start. They leave a spinner.

What breaks and how you notice

Transcoding falls behind. Alert on the age of the oldest job. Creators see "processing" longer.
An edge or region degrades. Players report stalls, so alert on the stall ratio per region. The player retries a second CDN URL.
Cold content floods the origin. Watch the regional hit ratio and origin requests. Pre-warm and collapse requests.
The catalogue cache fails. The database then gets 35,000 reads a second instead of 1,750. Serve stale copies, collapse misses and limit the rate.
The progress store is slow. Never make playback wait for it. Start from the start, or from the last position in the app.
A bad rendition. Check before publishing. The player skips a broken one.

Mid and Senior answers, and the follow-ups

A Mid answer covers the whole path: signed upload, queue, chunked parallel transcoding, segments in object storage, a manifest, adaptive bitrate and a CDN, with the bandwidth arithmetic. A Senior answer adds cost and tiers: popular versus long tail, request collapsing, pre-warming, edge placement, start-up and the failure handling above.

Expect these follow-ups:

Recommendations. Send watch events to a stream. Offline jobs build a list per user, stored in a cache. Never compute it on the home page request.
Search. Index titles and tags in a search engine. Update the index from catalogue changes through a queue, so search lags by seconds.
DRM (copy protection). Encrypt segments. A licence service gives keys only to authorised players. This adds one call to start-up.
Thumbnails. One more job in the same pipeline picks frames and stores a small image set in object storage, served by the CDN.
Live. Segments are made seconds ago for everyone. There is no time to prepare, so delay and the CDN fill dominate. That is a different question, designing live streaming, with its own numbers.

📌 Summary: Chunk and parallelise transcoding. Tier the CDN, collapse requests and pre-warm releases, since delivery is the cost. Start low and climb for a fast start. Alert on stalls, queue age and hit ratio.

What Meera learns next

Video is huge, but each viewer's needs are simple. Next: ride hailing. The data is small but moves every few seconds, and one rider must be matched to one driver.

Check your understanding

A transcoding job crashes after writing half its segment and the queue retries it on another worker. What makes the retry safe?
The edge answers 90% of 15 Tbps. A regional tier catches 80% of the misses. How much traffic does the origin see?
On a 4 Mbps link, which start gives a first picture in under 2 seconds?
A big series launches tonight and viewers arrive within minutes. What should you do beforehand?
The interviewer asks you to add "recommended for you". Where do you compute it?