Design video streaming (Netflix, YouTube)

Let creators upload videos and let tens of millions of viewers watch them any time. The hard parts are a parallel transcoding pipeline, adaptive bitrate playback from a manifest, and CDN delivery, which is both the main cost and the main speed lever.

Design a video streaming service where creators upload videos and viewers watch them any time on phones, TVs and browsers. Plan for 50 million viewers a day, about 5 million watching at the evening peak, and 100,000 hours of new video uploaded each day.

Requirements

Functional

A creator uploads a video file, and the upload resumes if the connection drops.
The service transcodes each upload into several qualities (renditions) that suit phones, TVs and browsers.
A viewer browses a catalogue, opens a video and watches it with adaptive quality, which means the player picks the quality from its measured speed.
Playback resumes where the viewer stopped, on any device, and a watch history is kept.

Non-functional

50 million viewers a day, each watching about 1 hour. About 5 million watch at the same time at the evening peak.
Playback starts in under 2 seconds for 9 in 10 plays (p90). Stalls (rebuffering) take under 1% of watch time.
Playback is 99.95% available. An upload may be a little less available.
A new video is playable in its lowest quality within about 2 minutes, and in every quality within about 10 minutes.
An uploaded original is never lost. Delivery cost per viewer-hour stays within budget.

Out of scope

Live streaming, where video is made seconds ago for everyone at once (see the live streaming write-up).
Recommendations and search (good follow-ups).
Copy protection (DRM), thumbnails, ads, subtitles and offline downloads.

Estimate

Viewing: 50M viewers x 1 hour = 50M viewer-hours a day. 50M / 24 = about 2M watching on average. Peak about 2.5 times that = about 5M at once.
Bandwidth: assume 3 Mbps average bitrate. 5M x 3 Mbps = 15M Mbps = 15 Tbps at the peak.
Daily delivery: 1 hour at 3 Mbps = 3 x 3,600 / 8 = 1,350 MB = 1.35 GB. 50M hours x 1.35 GB = about 70 PB a day (67.5 PB). At an assumed 1 cent per GB that is about $675K a day, so delivery is the main running cost.
Segment requests: 6-second segments, so 5M / 6 = about 830K requests per second at the edge. Each segment is 3 x 6 / 8 = 2.25 MB. Check: 830K x 2.25 MB = about 1.9 TB/s = 15 Tbps.
Catalogue reads: 50M x 20 page loads = 1B a day = about 11,600 per second on average, about 35K per second at peak. Plays started: 50M x 4 = 200M a day = about 2,300 per second, about 7K at peak.
Uploads: 100,000 hours a day at 10 minutes each = 600,000 videos a day. 600,000 / 86,400 = about 7 uploads per second. Original at 8 Mbps = 3.6 GB an hour, so 100,000 x 3.6 GB = about 360 TB a day.
Renditions: 0.4 + 0.8 + 1.5 + 3 + 6 = 11.7 Mbps for 5 qualities. One hour = 11.7 x 3,600 / 8 = about 5.3 GB. 100,000 hours = about 530 TB a day. New storage = 360 + 530 = about 0.9 PB a day, about 325 PB a year.
Pipeline: a 10-minute video = 100 segments of 6 s x 5 renditions = 500 jobs. 600,000 videos x 500 = 300M segment files a day = about 3,500 object writes per second.
Encoding compute: if one core encodes about as fast as the video plays, 100,000 hours x 5 renditions = 500,000 core-hours a day = about 21K cores busy all day, double that at an upload peak. Scale by queue depth.
Progress: a position report every 30 s per viewer = 5M / 30 = about 170K writes per second at peak. About 20 shards at 10K writes per second each.

API

POST /videos {title, sizeBytes} -> 201 {videoId, uploadUrls}
PUT <uploadUrl> (one part of the file, direct to object storage) -> 200 {etag}
POST /videos/{id}/complete {etags} -> 202 {status: processing}
GET /videos/{id} -> 200 {title, durationS, status}
GET /videos/{id}/playback -> 200 {manifestUrl (signed), resumeSeconds}
GET <manifestUrl> -> 200 manifest listing renditions and segments (served by the CDN)
GET <segmentUrl> -> 200 video bytes (served by the CDN)
PUT /me/progress/{videoId} {positionS} -> 204
GET /me/history?cursor= -> 200 {items, nextCursor}

Data model

videos(video_id PK, owner_id, title, duration_s, status [uploading|processing|ready|failed], created_at): SQL, replicated
renditions(video_id, name, bitrate_kbps, segment_count) PK(video_id, name): SQL
transcode_jobs(video_id, rendition, chunk, status, attempts): a job queue plus a status table that counts finished jobs per video
Object keys: originals/{video_id} and v/{video_id}/{rendition}/{n}.m4s and v/{video_id}/master.m3u8 (the manifest)
watch_progress(user_id, video_id) PK -> position_s, updated_at: key-value store, sharded by user_id, latest value overwritten
watch_history(user_id, watched_at, video_id): append-only, written through a stream
Cache: video:{id} -> catalogue JSON, expiry 5 minutes

A request, step by step

Upload: the creator's app calls POST /videos. The upload service creates a video row with status uploading and returns signed URLs, temporary links that allow one upload each. The app sends the file in parts of about 8 MB straight to the original store (object storage), and resends only missing parts after a drop. API servers never carry the bytes.
Transcode: on "complete", the upload service sets the status to processing and puts jobs on the queue: one per 6-second chunk and rendition, about 500 for a 10-minute video. Workers read the original, encode their chunk and write the segment to a fixed key in the segment store. When the lowest rendition is complete the video is playable. When all are done, a final job writes the manifest and the status becomes ready.
Browse: the app asks the catalogue service for pages of titles. The service answers from the catalogue cache (about 95% of requests) and from the replicated database on a miss. The cache entries expire after 5 minutes, so an edited title may look stale for that long.
Play: the app calls GET /videos/{id}/playback. The service checks the user, reads the resume position from the progress store and returns a signed manifest URL. The player fetches the manifest from the CDN edge, then segments one at a time, starting at a low rendition. A miss at the edge goes to the regional tier, and a miss there goes to the segment store, which keeps the copy at both layers.
Adapt and resume: after each segment the player measures its download speed and picks the next quality, the highest one under about 80% of that speed. Every 30 seconds it reports its position. The service overwrites the stored position and sends a watch event to a stream for history and recommendations.

Deep dives

The transcoding pipeline: one source into many segments, in parallel

Weak: one worker encodes the whole file in order. A 10-minute video with 5 renditions is about 50 minutes of work on one core, and a 2-hour film is about 10 hours. A crash at minute 40 restarts from zero, and one long job blocks the line behind it.

Good: chunk it and queue the jobs. Cut the original at keyframes (full pictures that later frames refer to) into 6-second chunks. A 10-minute video becomes 100 chunks x 5 renditions = 500 jobs on a queue. Workers run them in parallel, so the video is ready in minutes. A failed job is retried alone. Each job writes to a fixed key, so running twice is harmless (idempotent). After 5 failures a job goes to a dead-letter queue. A status table counts finished jobs per video.

Great: add priority and scaling. Use a fast lane for the lowest rendition so the video becomes playable early, and a slower lane for 1080p. Scale workers by queue depth and the age of the oldest job, because 21K cores is only an average and the evening may need double. Before the status becomes ready, check each rendition: the duration matches the original and every segment exists. Align cut points across renditions so the player can switch quality at any boundary.

Trade-off: smaller chunks give more parallelism but more overhead per job, and every chunk starts with a keyframe, so files grow a little. Six seconds is a common balance.

CDN delivery: the main cost and the main speed lever

Delivery is about 70 PB a day, so it decides both cost and speed. Assume the most-watched 1% of videos get about 80% of views. This is an assumption, but real catalogues are steep. The rest is the long tail: millions of videos each watched rarely.

Weak: serve segments from the origin store. 15 Tbps would overwhelm it, and every viewer's bytes would leave from one place, far from most of them.

Good: put a CDN in front. Popular segments stay cached at the edge. If the edge answers 90% of requests, 10% of 15 Tbps = 1.5 Tbps still reaches the origin. The long tail is the problem: a title watched once an hour is rarely in cache.

Great: add layers, collapse requests and pre-warm. Edge servers send misses to a regional tier, a larger shared cache. If it catches 80% of those misses, the origin sees 15 x 0.1 x 0.2 = 0.3 Tbps. Request collapsing means that when 1,000 viewers ask for the same new segment, the cache makes one request behind it and shares the answer. Before a big release, pre-warm the edge with the first segments of the new title, so the first viewers do not all trigger a cache fill. Evict the least recently used files first, since disk is limited.

Where to put edge servers: in large cities, and for the heaviest traffic inside big internet providers' networks, so video does not cross the wide internet. Trade-off: your own edge servers cost money to run and plan. Start with a CDN provider and add your own when volume pays for it.

Start-up time and adaptive bitrate

The goal is under 2 seconds for 9 in 10 plays. The playback call takes about 100 ms and the manifest about 100 ms. The rest is the first segment.

Weak: start at the highest quality. A 6-second segment at 6 Mbps is 4.5 MB. On a 4 Mbps link it takes 4.5 x 8 / 4 = 9 seconds. The viewer stares at a spinner.

Good: start low and climb. At 0.8 Mbps the first segment is 0.6 MB and takes 0.6 x 8 / 4 = 1.2 seconds. The total is 0.1 + 0.1 + 1.2 = about 1.4 seconds. After each segment the player computes speed = size x 8 / time and picks the highest rendition under 80% of it. It drops a step if less than 10 seconds of video is buffered.

Great: shorten the first segments and remember the speed. Use 2-second segments at the start. Remember the last measured speed on the device, fetch the manifest when the viewer opens the title page, and add a hysteresis rule (a small delay before switching up) so quality does not flip at every segment.

Trade-off: the first seconds look blurry and short segments mean more requests. Viewers accept a blurry start. They leave a spinner.

Resume position and watch history writes

Every viewer reports a position every 30 seconds, so 5M / 30 = about 170K writes per second at the peak.

Weak: write each report as a row in the main SQL database. One database handles about 2,000 simple writes a second. This would need about 85 of them, and it would slow the catalogue queries that share the database.

Good: a key-value store for the latest position. Key it by user and video, overwrite the value, and shard by user id. About 20 shards at 10K writes per second each cover the peak. A crash loses at most 30 seconds, which is acceptable.

Great: separate position from history. The position (a single overwritten value) goes to the key-value store. The history (an append-only record that recommendations use) goes through a stream and is written in batches, and only on pause, exit and every few minutes, so most reports never become history rows. Playback never waits for either write.

Trade-off: if a device is offline, the position is stale until it reconnects. Last write wins by the server's timestamp, so the newest report is kept.

What breaks

Transcoding falls behind after an upload spike. Alert on the age of the oldest job and scale workers by queue depth. Creators see "processing" for longer, but playback of existing videos is not affected.
An edge or region degrades. Players report stalls, so alert on the stall ratio per region. Players retry a second CDN URL, and traffic weights shift to healthy edges.
A new title floods the origin because the caches are cold. Watch the regional hit ratio and origin requests. Pre-warm before releases and collapse repeated misses.
The catalogue cache fails and 35K reads per second hit a database that handles about 2K. Serve stale copies, collapse misses and limit the rate so the database survives.
The progress store is slow or down. Playback never waits for it: the player starts from the beginning or from the last position it holds locally.
A bad rendition is published. Check duration and segment count before the status becomes ready. The player skips a rendition that fails to load.

What each level is expected to cover

Mid level: Covers the whole path: signed resumable upload, a queue of chunked transcode jobs, segments in object storage, a manifest, adaptive bitrate and a CDN. Does the bandwidth arithmetic (about 15 Tbps at 5M viewers) and says delivery dominates storage.

Senior: Adds cost and tiers: popular versus long tail, an edge plus regional tier with request collapsing, pre-warming and edge placement. Quantifies start-up time, separates position from history writes, sizes the transcoding fleet by queue depth, and names the failure signals.

Common mistakes

Sending video bytes through application servers instead of a signed upload to object storage and a CDN for playback.
Transcoding a whole file on one machine, so the creator waits an hour and a crash restarts the work.
Having the server choose the quality, instead of the player choosing the next segment from its measured speed.
Treating storage as the main cost and forgetting that delivery moves about 75 times more data each day.
Writing every position report to the main database, or letting playback wait for the progress write.
Skipping the long tail: assuming a CDN hit ratio of 99% for a catalogue where most titles are rarely watched.

Follow-up questions

How would you add recommendations? (Watch events to a stream, offline jobs, a ready list per user in a cache.)
How would you add search over titles and tags? (A search index fed by catalogue changes through a queue.)
How would you add copy protection? (Encrypt segments, a licence service gives keys to authorised players, one more call at start-up.)
How would you make thumbnails and hover previews? (One more job in the same pipeline, images in object storage, served by the CDN.)
How would this change for a live event? (Segments are made seconds ago for everyone, so there is no time to prepare and delay dominates.)
How would you cut the delivery bill by 20%? (Better hit ratio, a lower top bitrate, more efficient video formats, edge servers inside internet providers.)