Design Dropbox or Google Drive

Dropbox and Google Drive keep a folder in sync across all your devices and let you share files with others. The interesting problems are uploading only what changed, keeping metadata consistent, notifying devices quickly, and resolving conflicts when two devices edit the same file offline.

Requirements#

Functional

• Upload, download, rename, move and delete files and folders.
• Changes on one device appear on the user's other devices.
• Share files and folders with other users.
• File version history.

Non-functional

• Never lose a file, even if a device or a datacenter fails.
• Sync large files efficiently, including over slow connections.
• 500 million users with tens of petabytes of data.

Chunking#

Files are split into blocks (for example, 4 MB each), and each block is identified by the hash of its content.

• Delta sync: when a file changes, only the blocks whose hash changed are uploaded. Editing one slide in a 200 MB deck uploads a few MB.
• Deduplication: identical blocks (the same installer uploaded by a million users) are stored once.
• Resumable uploads: a failed upload continues from the last finished block.

Architecture#

1. The client notices a change, chunks the file and asks the block service which hashes are missing.
2. It uploads only the missing blocks.
3. It commits a new file version to the metadata service: an ordered list of block hashes.
4. The metadata service publishes a change, and the user's other devices are told to sync.

Metadata#

The metadata must be strongly consistent: a file's version list or a folder rename can't be half applied. A relational database sharded by namespace (user or shared folder) works well.

files    (id, namespace_id, path, latest_version, is_deleted)
versions (file_id, version, block_hashes, size, created_by, created_at)
blocks   (hash, size, storage_key, ref_count)

Each namespace has a journal of changes with increasing numbers. A device stores the last journal number it has applied and asks "what changed after 8,241?", which makes sync incremental and restartable.

Notifying devices#

Clients hold a long-poll or WebSocket connection to the notification service. When their namespace changes they get a ping and then pull the journal. Pings carry no data, so a missed one is harmless: the next sync catches up.

Conflicts#

If two devices edit the same file from the same base version, the first commit wins. The second commit fails its version check, and the client saves its copy as report (Bob's conflicted copy).docx. Silent merging is risky for binary files, so keeping both is the safe default.

Durability#

• Blocks are replicated across zones, or erasure coded (for example 10 data + 4 parity) to tolerate failures at lower cost than three full copies.
• Blocks are reference counted; a block is deleted only when no version uses it, after a grace period.
• Old versions are kept for 30 days or more, which doubles as protection against ransomware.