Sync, conflicts, sharing and the bill

Part of the free System Design course on LogicWiz, in the chapter: Design file storage and sync (Dropbox, Google Drive). The interactive whiteboard exercises and simulations run in the app.

Week 18, Day 3: The edit that vanished

"I edited the budget on the plane. When I landed, my manager's change from the same hour had replaced my whole afternoon."

Meera, retelling a sync bug to Kabir

Sync: from a changed file to a refreshed laptop

Walk one change end to end. The operating system tells the app that a file in the synced folder changed. The app waits two seconds, because editors save in several small writes. It then cuts the file into chunks, hashes them, compares with the last synced list, and uploads only the chunks that differ. Then it commits.

Now the other devices. Three options:

Weak: poll the whole folder listing. With 5 million devices online and a poll every 10 seconds, that is 500,000 requests a second. Each one lists thousands of files, and nearly every answer is "nothing changed".

Good: poll with a cursor. The device sends "changes since 1840". The server reads only log rows after 1840, and the answer is tiny. But 5M / 10 = still 500,000 requests a second.

Great: ping, cursor and a slow backup poll. The notification service pings a device only when its user has a new change: about 700 pings a second at peak, a little more with shared folders. The device then asks for changes since its cursor. A backup poll every 5 minutes covers lost pings: 5M / 300 = about 17,000 cheap cursor checks a second. The ping is a hint, never the data.

A laptop closed for a week sends its old cursor and receives every change since. If the cursor is older than the log is kept, the client compares whole folder trees instead. Applying a change twice must be harmless, so a repeated message does no damage.

Conflicts: keep both copies

Both laptop and phone hold report.xlsx at version 7. The laptop goes offline on a flight and edits. Meanwhile the manager's phone commits version 8. When the laptop lands, it commits with base version 7. The server sees the current version is 8, not 7. That is a conflict: two edits grew from the same parent.

Weak: last writer wins. The later commit replaces the earlier one. One person's afternoon vanishes without a warning. Worse, "later" is judged by device clocks, which can differ by minutes, so even by its own rule it can pick the wrong winner.

Good: refuse and ask. Reply 409 and let the user choose. Safe, but a sync client runs in the background, and a laptop may sit closed for days.

Great: keep both copies. The first commit becomes version 8. The second device's file is saved beside it as "report (Meera's laptop, conflicted copy).xlsx". Both appear on every device, and a person merges them. The cost is some clutter and a few extra chunks, because most chunks are shared. Nothing is lost and no clock is needed.

The check is one statement inside the commit transaction: update the file's current version to the new number only where it still equals 7. If no row changes, return 409. This is optimistic concurrency: assume no clash, and verify at commit.

Meera asked, "Can the server merge them?" For a spreadsheet or image, there is no safe general merge. Live co-editing is its own problem and out of scope.

Write the rules down.

Sharing links and access checks

A share link carries a random token of 16 bytes, shown as about 22 characters. Made with a secure random generator, a token with 2^128 possible values cannot be guessed in practice. The links row holds the token, the file, a role (view or edit) and an expiry. A request for a link goes to the metadata service, which checks that the link is live and then returns download links.

Sharing a folder with a person means a permission row on the folder:

Weak: copy the permission onto every file inside. A folder of 100,000 files needs 100,000 writes to share and again to revoke, and moving a file breaks it.

Good: store it on the folder and walk up the tree. The check looks at the file, then its parents, until it finds a grant. Cache the answer for up to 60 seconds, and clear that cache entry when a grant is revoked.

Great: the same, checked on every request, plus short-lived download links. The metadata service decides; the client and the storage never do. Download links last 10 minutes. After a revoke, new requests fail at once, but a link already issued works for up to 10 minutes. Say that trade-off aloud.

Trash, versions and the bill

Delete moves a file to trash (a deleted_at time). It can be restored for 30 days. Then a purge removes the rows and lowers each chunk's reference count. A cleanup worker removes chunks at zero, but waits 24 hours first, because an upload in progress may have just matched that chunk.

Versions are cheap because a version is only a list of hashes. A 200 MB deck edited 10 times with 8 MB changed each time costs 200 + 10 x 8 = 280 MB. Full copies would cost 11 x 200 = 2,200 MB.

Back to the bill from lesson 1: 5.25M GB at 2 cents is about $105,000 a month. Assume 60% of bytes were not read for 90 days. An archive tier at half the price saves 3.15M GB x $0.01 = about $31,500 a month, near 30% of the bill. The trade-off is a slower first read of a cold file, and often a retrieval fee.

What breaks, and what each level says

Chunks arrive but the commit never does. Stray chunks pile up. Watch the volume of unreferenced bytes; the cleanup worker removes them.
Pings are lost. Devices lag until the backup poll. Watch the time from commit to "applied on device", at the 95th percentile.
One metadata shard is slow. Users on it cannot browse. Replicas and a cache help. Alert on shard latency.
A chunk is damaged. The client checks the hash on download, fetches again, and raises an alert.

A Mid answer gets the numbers, chunks, two stores and a notification path. A Senior answer adds keep-both conflicts with the version check, the ping-plus-backup-poll numbers, permission checks that walk up the tree, and the cost levers.

Expect these follow-ups. "A shared folder holds two million files?" Store the grant on the folder and cache. "Inserts shift every chunk?" Use content-defined cutting. "Dedup across users leaks information?" Dedup within an account, or ask for proof the client holds the bytes.

📌 Summary: Sync with a cursor, a ping and a slow backup poll. On conflict, keep both copies, never last-writer-wins. Check access on every request, and count versions and cold data in the bill.

What Meera learns next

Next question: a web crawler. Like sync, it must avoid fetching the same bytes twice, and a content hash helps again. But no client cooperates. The other side is the open web, and a crawler must be polite, fast and never get stuck.

Check your understanding

Five million devices are online. Compare polling "changes since my cursor" every 10 seconds with a ping plus a backup poll every 5 minutes. About how many requests a second does each cost?
A laptop commits with base version 7, but the server is already at version 8. What should happen?
A candidate says: "Use the timestamp. The newest edit wins, and we avoid conflict copies." What is the strongest objection?
A share link is revoked. Two minutes earlier, a 10-minute download link was issued from it. What happens next?
A chunk's reference count reaches zero. Why does the cleanup worker wait 24 hours before deleting it?