Move 100 petabytes from the US East Coast to the West Coast

01The design, in the six-step framework

1. Requirements

Functional.

• Move 100 PB of stored data (objects in a storage system) from a data centre on the US East Coast to one on the West Coast.
• The source keeps serving reads and writes during the move; nothing goes down.
• Every byte must be verified at the destination; the move ends with a cut-over after which the west copy is authoritative.
• Out of scope: application changes beyond repointing to the new location; buying the second data centre.

Non-functional.

• Deadline: complete within 90 days.
• Network: a dedicated 100 Gbit/s link exists, shared with production traffic that needs about 40 percent of it in the daytime.
• Data changes: about 0.5 percent of objects change or are added per day (500 TB per day of churn).
• Integrity: zero undetected corruption; a checksum per object and per batch.

2. Core entities

• Object: key, size, content hash, version, last_modified, location bitmap (east, west, both).
• Manifest: a snapshot of the object listing at a moment: key, version, hash; the unit of planning and of verification.
• Batch: a set of objects assigned to one appliance or one network stream; state (loaded, in transit, ingested, verified).
• Change record: key, operation, version, time; the churn since the manifest was taken.
• Verification result: batch, objects checked, mismatches, re-queued keys.

3. API

Mostly internal control-plane operations, plus the storage system's existing API.

• POST /manifests → snapshot the listing (a scan of the metadata store) and store it as a sorted file; GET /manifests/{id}/diff?since= → keys changed since.
• POST /batches {manifest, key range, transport: appliance | network} → batch id; POST /batches/{id}/state as it moves.
• POST /verify {batch} → compares destination hashes to the manifest and returns mismatches.
• POST /cutover {phase} → flips the location-routing policy: reads west, writes both, then writes west only.

4. High-level design

The plan, in phases:

1. Snapshot. Take manifest M0 of all 100 PB (keys, versions, hashes). From this moment, record every write in a change log.
2. Bulk by road. Partition M0 into batches of about 1 PB by key range; load each onto an appliance (local copy at disk speed, 1 PB in roughly a day across many drives), ship, ingest at the west side, verify hashes against M0, mark the batch done.
3. Churn by wire. Continuously replay the change log over the network into the west; this is 500 TB a day, about half the usable link, and keeps the west copy within hours of the east.
4. Converge. When all batches are verified, take manifest M1 on both sides and diff; copy the stragglers over the network; repeat until the diff is tiny.
5. Cut over. Writes go to both sides (dual-write) while reads move west region by region; after a quiet period with zero diff, writes go west only and east becomes a read-only replica, then is retired.

5. Deep dives

Road or wire: how do we decide, and is it really both?

The arithmetic decides: the link cannot carry 100 PB plus churn in 90 days, and the churn alone is a steady 46 Gbit/s, so the bulk goes by appliance and the wire is reserved for changes and for the final convergence. The cost comparison is also stark: a dedicated 100 Gbit/s cross-country circuit for three months costs more than a fleet of rented appliances. The honest caveat is operational: appliances need people to load, drive, receive and ingest them, a chain of custody with encryption at rest (keys travel separately), and a plan for a lost or damaged unit (re-ship the batch; nothing is deleted from the east until verified in the west). Capacity planning here is throughput of a process with humans in it, and the batch size is chosen so that one loss costs a week, not a month.

How do we verify 100 PB without reading it twice over the wire?

Hashes travel with the data and with the manifest. Every object already has a content hash in the metadata store (if it does not, computing it during the appliance load is free because the bytes are being read anyway). The west side recomputes the hash on ingest and compares with the manifest; a mismatch re-queues the object for the network path. A batch is done only when every object in its key range verifies. On top of per-object hashes, a per-batch Merkle-style digest lets the two sides agree that a whole range matches with one small exchange, which is what makes the convergence diffs cheap: compare digests of key ranges, descend only into ranges that differ. Verification reads the data once at each end, at disk speed, never over the link.

How does the source stay live and the cut-over stay safe?

The source never stops; the change log captures everything after the snapshot, keyed by object and version, so a replay is idempotent (a version is written once; an older version arriving late is ignored). Deletes are recorded too and replayed as deletes. During dual-write, a write that succeeds east and fails west is retried from the log, so the west is eventually complete; reads from the west during this window may miss the last few minutes of writes, which the product must accept or route "read your own recent writes" to the east. Cut-over is reversible at every step until the east is retired: the routing policy is a staged rollout with a rollback path, moved region by region with error rates watched.

What does the program plan look like for a TPM-style interviewer?

Milestones with numbers: week 1 snapshot and first appliances loaded; weeks 2 to 11 ten appliances per week in flight, two days of buffer per trip; churn replication running from week 1 with lag under 6 hours as the health metric; week 12 convergence under 1 TB of diff; weeks 13 to 14 cut-over by region; week 15 east read-only, week 16 decommission start. Risks and their mitigations: appliance loss (re-ship, nothing deleted), link contention with production (replication throttled by a daytime cap), hash mismatches above 1 in 10⁹ (stop and investigate, it is a firmware or memory problem), and the human one, a key range that keeps changing faster than the wire can carry it (identify hot prefixes early and move them last, right before cut-over).

6. What a strong answer sounds like

• Do the link arithmetic out loud and reach the conclusion yourself: 93 days at full link, so the bulk goes by road.
• Treat churn as its own stream with its own bandwidth requirement, and keep the wire for it.
• Describe verification as hashes carried in manifests, compared at disk speed, with range digests for cheap convergence.
• Make the cut-over a staged, reversible routing change, and say what reads see during dual-write.
• Expected follow-ups: encryption and chain of custody for the appliances; what if the deadline were 30 days (more appliances in parallel, loading is the limit); how to prove to an auditor that nothing was lost (manifest diff equals zero, signed).
• Common mistake: proposing compression or deduplication as the fix for the bandwidth gap without checking whether the data compresses at all.

02Where this question was reported

• IGotAnOffer · TPM list
• IGotAnOffer · Google TPM list