File storage and sync (Dropbox, Google Drive)

Store large files safely and keep a folder equal on every device. The bytes live in object storage as hashed chunks. The file tree, versions and sharing live in a metadata database. Sync runs on a change log with a cursor, and conflicts keep both copies.

Design a file storage and sync service like Dropbox or Google Drive for 50 million users who store about 5 petabytes in total, where a change on one device appears on their other devices within seconds.

Requirements

Functional

Upload, download and delete files and folders. A single file can be up to 10 GB.
Sync a chosen folder across a user's devices (laptop, phone, web), including edits made while offline.
Share a file or folder with a named person or through a link, with a view or edit role.
Keep old versions and a trash for 30 days, so a user can restore a file.

Non-functional

Scale: 50 million registered users, 10 million active in a day, about 5 PB stored and 5 billion files.
Sync delay: a change appears on the user's other devices within 10 seconds for 95 out of 100 changes.
Durability: once the app says a file is saved, it survives disk and machine failures. Several copies of every chunk are kept.
Availability: 99.9% for upload and download. Files already on a device stay usable offline.
Efficiency: a broken upload resumes, and an edit sends only the chunks that changed.

Out of scope

Live co-editing of one document with cursors (a different problem with its own algorithms).
Full-text search inside files, virus scanning and billing plans (mention them, do not design them).

Estimate

Stored data: 50M users x 100 files x 1 MB average = 5,000 TB = 5 PB. File count: 50M x 100 = 5 billion.
Changes: 10M active users x 2 changed files = 20M a day. 20M / 86,400 s = about 230 a second. Peak is about 3 times that: 700 a second.
Upload bandwidth: 20M x 1 MB = 20 TB a day. 20 TB / 86,400 s = about 230 MB a second, roughly 1.8 Gbps. At peak: 700 MB a second, about 5.6 Gbps.
New bytes: skipping chunks the server already holds, assume about a quarter of uploaded bytes is new. That is 5 TB a day, and 5 TB x 365 = about 1.8 PB a year.
Chunks: 4 MB each. A 2 GB file is 2,048 / 4 = 512 chunks. A 200 MB deck is 50 chunks, so a one-slide edit sends 4 to 8 MB.
Metadata: 5 billion files x about 300 bytes = 1.5 TB. The chunk index adds roughly 0.5 TB (billions of unique chunks at about 100 bytes a row, plus indexes). Total 2 TB, so 8 shards of 250 GB each (sharded by user id).
Metadata reads: 10M users x 20 folder listings = 200M a day = about 2,300 a second. Peak: about 7,000 a second, mostly served by a cache.
Connections: 10M users x 2 devices = 20M devices. A quarter online at peak = 5M open connections. At 100,000 per server, about 50 notification servers.
Cost: assume object storage at 2 cents per GB per month. Sharing identical chunks saves about 30%: 3.5 PB. Extra copies for safety add about 50%: 5.25 PB. 5.25M GB x $0.02 = about $105,000 a month.

API

POST /v1/uploads {path, size, chunk_hashes[]} -> 200 {upload_id, missing[]}
PUT /v1/uploads/{upload_id}/chunks/{hash} (bytes) -> 204
POST /v1/uploads/{upload_id}/commit {base_version} -> 201 {file_id, version} (409 if the file moved on: a conflict)
GET /v1/files/{file_id}/versions/{version} -> 200 {size, chunks[{hash, url}]}
GET /v1/changes?cursor=1840 -> 200 {changes[], next_cursor}
GET /v1/folders/{folder_id}/children -> 200 {items[]}
POST /v1/files/{file_id}/links {role, expires_at?} -> 201 {url}
DELETE /v1/files/{file_id} -> 204 (moves to trash; POST /v1/files/{file_id}/restore brings it back)

Data model

files(file_id PK, owner_id, parent_id, name, current_version, deleted_at): the tree; sharded by owner_id.
versions(file_id, version, size, chunk_hashes[], device_id, created_at): a version is an ordered list of chunk hashes.
chunks(hash PK, size, ref_count): the chunk index; bytes live in object storage under the hash.
changes(user_id, seq, file_id, kind): the change log; seq grows by one per change for each user.
permissions(file_id, principal_id, role) and links(token PK, file_id, role, expires_at): who may open what.

A request, step by step

Save: the client app notices a changed file, waits two seconds, cuts it into 4 MB chunks and hashes each one. It sends the hash list to the metadata service.
The metadata service looks each hash up in the chunk index and answers with the missing ones. The client sends only those, up to four at a time, through the upload service into object storage.
Commit: the client calls commit with its base version. In one transaction the service checks the file is still at that version, writes the new version row, updates the current version and adds a change-log row. If the version moved on, it returns 409 and keeps both copies.
Notify: the commit puts a change event on the queue. The notification service pings the user's other online devices. The ping carries no data.
Sync: each device asks for changes since its cursor, gets the new entries, downloads only the chunks it lacks, checks the hashes and updates its cursor. A backup poll every 5 minutes covers lost pings.
Download: the metadata service checks permission and returns the chunk list with short-lived links. The client fetches chunks in parallel and joins them.
Delete: a file moves to trash. After 30 days the purge lowers chunk reference counts, and the cleanup worker removes chunks at zero after a further 24 hours.

Deep dives

Chunking, hashing and de-duplication

Weak: upload the whole file in one request. A 2 GB file needs one unbroken connection. A drop at 97 percent restarts it, and a one-slide edit resends 200 MB.

Good: fixed-size chunks. Cut every file into 4 MB chunks. A 2 GB file is 512 chunks, so a drop after 358 chunks leaves 154 to send. A 200 MB deck is 50 chunks, and an edit made in place changes one or two. Chunks can also go in parallel, four at a time.

Great: chunks named by their content hash. The client hashes each chunk with SHA-256 (32 bytes) and sends the list first. The server answers which hashes it lacks. This gives resume, delta upload and de-duplication in one step. Identical chunks are stored once with a reference count, which saves about 30% of 5 PB in this estimate. A renamed or copied file uploads nothing.

Trade-offs: fixed cuts shift when bytes are inserted near the start of a file, so every later chunk changes. Some formats, such as zipped office files, may rewrite much of the file on save, so many chunks change anyway. Content-defined cutting fixes that at the cost of more client work. Smaller chunks mean more metadata and requests. Larger chunks resend more. Dedup across users can reveal that another person holds a file, so many designs dedup only inside one account.

Syncing devices with a cursor and a ping

Weak: every device polls the full folder listing. With 5M devices online and a poll every 10 seconds, that is 500,000 requests a second. Each lists thousands of files, and nearly every answer is "nothing changed".

Good: poll with a cursor. The change log numbers every change per user. A device sends its cursor (the last change it applied) and receives only newer entries. Each answer is tiny. But at one poll per 10 seconds it is still 500,000 requests a second.

Great: ping, cursor and a slow backup poll. The notification service keeps an open connection per online device. After a commit it pings only that user's other devices: about 700 pings a second at peak. The device then asks for changes since its cursor. A backup poll every 5 minutes covers lost pings: 5M / 300 = about 17,000 cheap cursor checks a second, served from the cache. The ping is a hint, never the data.

Details that matter: applying a change twice must be harmless. A device offline for a week sends its old cursor and gets all changes since. If the cursor is older than the log is kept, the client compares whole trees. Measure the 95th percentile time from commit to applied-on-device.

Two devices edit the same file: keep both copies

Laptop and phone hold version 7. The laptop edits offline. The phone commits version 8. The laptop then commits with base version 7. The server sees the file is already at 8: a conflict.

Weak: last writer wins. The later commit replaces the earlier. One edit vanishes without a warning. "Later" is judged by device clocks, which can differ by minutes, so the rule can even pick the wrong winner.

Good: refuse and ask. Return 409 and let the user choose. This loses nothing, but a sync client runs in the background and a laptop may stay closed for days, so the decision waits.

Great: keep both copies. The first commit becomes version 8. The second edit is saved next to it as a conflicted copy, for example "report (Meera's laptop, conflicted copy).xlsx". Both reach every device and a person merges them. Because most chunks are shared, the extra storage is small. No clock is needed.

How the check works: inside the commit transaction, set the current version to the new number only where it still equals the base version. If no row changes, return 409. This is optimistic concurrency: assume no clash, and verify at commit. Automatic merge is only safe for some text formats, so offer it as a later feature.

Sharing, trash, versions and cost

Sharing links. A link carries a random 128-bit token (16 bytes, about 22 characters) from a secure random generator, which cannot be guessed in practice. The links row holds the token, file, role and expiry. For people, store a permission on the folder and walk up the tree at check time. This is good. A weak design copies the permission onto every file: a folder of 100,000 files means 100,000 writes to share and again to revoke. A great design adds a cache of up to 60 seconds (cleared when a grant is revoked) and checks on every request, with download links that last 10 minutes. After a revoke, new requests fail at once and an issued link works until it expires. State that trade-off.

Trash and versions. Delete sets deleted_at, and a file is restorable for 30 days. A version is only a list of hashes, so versions are cheap. Take a 200 MB deck edited 10 times, with 8 MB changed each time. It costs 200 + 10 x 8 = 280 MB. Full copies would cost 11 x 200 = 2,200 MB. After the purge, chunk reference counts drop, and a cleanup worker removes chunks at zero after 24 hours, because an upload in progress may have just matched one.

Cost. The bill is about $105,000 a month. If 60% of bytes were not read for 90 days, an archive tier at half the price saves 3.15M GB x $0.01 = about $31,500 a month. The trade-off is a slower first read of a cold file, and often a retrieval fee.

What breaks

Chunks arrive but the commit never does, so stray chunks pile up. Watch the volume of unreferenced bytes. The cleanup worker removes them after 24 hours, and the old version stays visible meanwhile.
Pings are lost or the notification servers restart. Devices lag until the backup poll, at most 5 minutes. Alert on the 95th percentile of commit-to-applied time, and let clients reconnect with jitter so 5M devices do not return at once.
One metadata shard is slow or down. Users on it cannot browse or commit, while other shards are fine. Replicas and the cache absorb reads. Alert on shard latency and fail over the replica.
A chunk is damaged on disk. The client checks the hash on download and fetches again from another copy, and the system raises an alert to repair the copy.
A client bug floods the commit API. Rate-limit per user and per device, and make commits idempotent (the same upload id commits once).

What each level is expected to cover

Mid level: A Mid candidate gets the numbers right: about 5 PB, 5 billion files and 700 changes a second at peak. They split bytes from metadata, cut files into chunks with hashes, and explain resume and delta upload. They add a notification path for other devices and say what happens on a conflict.

Senior: A Senior candidate also compares polling, cursor and ping-plus-backup with request rates. They keep both copies on a conflict using a base version check, and explain why last-writer-wins loses data. They design permission checks that walk up the tree, choose cache and link lifetimes with the revoke trade-off, and use cost levers such as dedup, cold tiers and version retention.

Common mistakes

Saying "put the files in object storage" and never explaining how another device learns of a change, or who may open a file.
Sizing the metadata database for petabytes instead of for the number of files (about 1.5 TB).
Uploading whole files, so a broken connection or a one-slide edit costs the full size again.
Resolving conflicts by last-writer-wins with timestamps, which silently loses an edit and trusts device clocks.
Making a new version visible before all its chunks are stored, so other devices can download a half-built file.
Copying permissions onto every file in a shared folder, which makes a share or revoke into hundreds of thousands of writes.

Follow-up questions

A user inserts a paragraph at the start of a large file and every fixed chunk changes. How would you reduce what is sent?
A shared folder holds two million files. How does the access check stay fast?
How would you stop de-duplication from revealing that another user holds a given file?
How would you add real-time collaborative editing, and what would change in this design?
How would you run this in two regions, and where would the metadata for each user live?