File storage and sync: requirements and numbers

Part of the free System Design course on LogicWiz, in the chapter: Design file storage and sync (Dropbox, Google Drive). The interactive whiteboard exercises and simulations run in the app.

Week 18: The 2 GB upload that started over

"The upload reached 97 percent, the train went into a tunnel, and the app began again from zero."

Meera, testing a file app on her commute

What "design Dropbox" really asks

Kabir opens the mock interview with one line: "Design a service like Dropbox or Google Drive." Meera answers fast. "Files go to object storage, users log in, done."

Object storage is a service that keeps files as named blobs, cheaply and with several copies for safety. It is the right home for the bytes. But Kabir stops her. "Your laptop and your phone hold the same folder. You edit a file on the laptop. How does the phone find out?"

That question is the whole problem. A drive product has three jobs:

  • Store: keep a folder tree of files safely, including very large ones.
  • Share: let other people open a file, with the right permission.
  • Sync: keep the copies on every device equal, even after one was offline.

Storing bytes is the easy part. Knowing what changed, on which device, and who may see it is the hard part.

A candidate in an earlier mock said, "It is a big bucket with a login." The interviewer asked one more question: "Two phones edit the same photo caption offline. Which one wins?" The bucket had no answer. Keep that question in mind. The last lesson answers it.

Questions to ask first

In the first five minutes, ask these. Each answer will earn a box on the board later.

  • Which clients? A desktop folder that syncs, a phone app and a web page.
  • How big can one file be? Say up to 10 GB.
  • Can users share with other people, by name and by link?
  • How long do we keep old versions and deleted files? Say 30 days.
  • Do two people edit one document live? Say no: that is a different problem.
  • Is a few seconds of delay acceptable before another device sees a change? Say yes.

Requirements, with numbers

Functional: upload, download and delete files and folders up to 10 GB. Sync a folder across a user's devices, including edits made offline. Share a file or folder with a person or by link. Keep versions and a trash for 30 days.

Non-functional:

  • Scale: 50 million registered users, 10 million active in a day.
  • Sync delay: a change shows on the user's other devices within 10 seconds for 95 out of 100 changes.
  • Durability: once the app says "saved", the file survives disk and machine failures, because several copies exist.
  • Availability: 99.9% for upload and download.
  • Efficiency: a broken upload resumes, and an edit sends only the part that changed.

Out of scope: live co-editing, search inside file contents, virus scanning and billing plans.

The estimate, step by step

Work from users to bytes. Say each step aloud.

  • Stored data: 50M users x 100 files x 1 MB average = 5,000 TB = 5 PB (petabytes). Files: 50M x 100 = 5 billion.
  • Changes: 10M active users x 2 changed files = 20M a day. 20M / 86,400 seconds = about 230 a second. Peak is about 3 times that: 700 a second.
  • Upload bandwidth: 20M x 1 MB = 20 TB a day. 20 TB / 86,400 = about 230 MB a second, roughly 1.8 Gbps. At peak: 700 MB a second, about 5.6 Gbps.
  • New bytes: after we skip chunks the server already holds (next lesson), assume about a quarter of uploaded bytes is new: 5 TB a day. 5 TB x 365 = about 1.8 PB a year.
  • Metadata (the file tree, versions and chunk lists): 5 billion files x about 300 bytes = 1.5 TB.
  • Tree reads: 10M users x 20 folder listings = 200M a day = about 2,300 a second. Peak: about 7,000 a second.
  • Open connections: 10M users x 2 devices = 20M devices. A quarter are online at peak: 5 million connections. One server holds about 100,000, so about 50 servers.
  • Cost: assume object storage at 2 cents per GB per month. Sharing identical chunks saves about 30%: 3.5 PB. Extra copies for safety add about 50%: 5.25 PB. That is 5.25M GB x $0.02 = about $105,000 a month.

Read the shape of the load. By bytes, the system is write-heavy: users upload more than they re-download. By requests, it is read-heavy: folder listings and change checks arrive about 10 times more often than commits (7,000 against 700 at peak). Say both. The first shapes object storage and bandwidth. The second shapes the metadata cache and the database.

Meera's first instinct was to size the database for 5 PB. Kabir pointed at the 1.5 TB line. The database holds descriptions of files, not the files. That one number is why the design has two stores.

Now write your own estimate. Use the same method with your own words.

The API and the data model

A chunk is one fixed-size piece of a file; the next lesson explains why we cut files up. Its hash is a short fingerprint of its bytes. With those two words, the calls are:

POST /v1/uploads {path, size, chunk_hashes[]} -> 200 {upload_id, missing[]}
PUT /v1/uploads/{upload_id}/chunks/{hash} (bytes) -> 204
POST /v1/uploads/{upload_id}/commit {base_version} -> 201 {file_id, version}  (409 on conflict)
GET /v1/files/{file_id}/versions/{version} -> 200 {size, chunks[{hash, url}]}
GET /v1/changes?cursor=1840 -> 200 {changes[], next_cursor}
POST /v1/files/{file_id}/links {role, expires_at?} -> 201 {url}
DELETE /v1/files/{file_id} -> 204  (moves to trash)

files(file_id PK, owner_id, parent_id, name, current_version, deleted_at)
versions(file_id, version, size, chunk_hashes[], device_id, created_at)
chunks(hash PK, size, ref_count)
changes(user_id, seq, file_id, kind)
links(token PK, file_id, role, expires_at)

Two kinds of call appear. Some move bytes (the PUT of a chunk). Others move descriptions (everything else). The changes table is a log with one row per change, and seq grows by one for each user. Sync will run on it.

flowchart LR
  C["Client"] --> M["Metadata: tree, versions, sharing"]
  C --> B["Bytes: chunks in object storage"]

📌 Summary: Size files by count and by bytes: 5 PB of bytes, 5 billion files, about 700 changes a second at peak. Ask about file size, sharing, versions and sync delay first.

What Meera learns next

Next she cuts files into chunks, fingerprints each one, and draws the first design. The 2 GB upload will never start from zero again.

Check your understanding

  1. In a mock, a candidate hears "design Dropbox" and answers: "Store files in object storage, add a login, done." What is the biggest gap?
  2. Ten million active users change two files each per day, at 1 MB per file. About what is the average upload bandwidth?
  3. A teammate sizes the metadata database for the full 5 PB of user data. What is wrong with that?
  4. Which of these is a requirement an interviewer can test?