Distribute large ML model files across a serving fleet

01The design, in the six-step framework

1. Requirements

Functional.

• A new model version (one or more files totalling 100 GB) is published; it must be present and verified on 10,000 serving machines before serving switches to it.
• Machines pull the version they are told to run; several versions may coexist on a machine; old versions are removed when no longer referenced.
• Publish, roll out by percentage, and roll back by pointing machines at the previous version.
• Out of scope: training, the serving runtime itself, model evaluation.

Non-functional.

• Fleet-wide distribution of 100 GB to 10,000 machines in under 20 minutes.
• Network: the origin bucket must not be the bottleneck; per-rack and cross-region links must stay under 50 percent utilisation.
• Integrity: a machine never loads a corrupted or partial file; every byte is checksum-verified before use.
• Availability of the distribution system 99.9 percent; a failure must not stop serving of the current version.

2. Core entities

• Model version: name, version id, manifest hash, total size, created_at, state (published, rolling, serving, retired).
• Manifest: version id, list of files, each file's list of 64 MB chunks with their content hashes, signature by the publisher.
• Chunk: content hash, size, location set (origin, regional caches, peers that have it).
• Machine state: machine id, rack, region, versions present (complete, verified), versions wanted, disk free.
• Rollout: version id, target percentage or cohort, started_at, health gates, rollback target.

3. API

• POST /versions {files in object storage} → builds the manifest (chunk hashes), signs it, returns version id. Called by the training pipeline.
• POST /rollouts {version, percent, cohorts} and POST /rollouts/{id}/rollback → the control plane; machines learn their wanted version by polling or push.
• Machine agent: GET /machines/{id}/desired → wanted versions; POST /machines/{id}/state → what it has, verified.
• Chunk transfer: GET /chunks/{hash} on origin, regional caches and peer agents; GET /chunks/{hash}/locations from the tracker.

4. High-level design

A rollout, end to end:

1. The training pipeline uploads files to the origin; the registry chunks them, computes a hash per chunk and a hash of the manifest, signs the manifest, and marks the version published.
2. An operator (or an automated pipeline) starts a rollout at 1 percent. The controller sets the desired version on 100 machines across racks and regions.
3. Each agent fetches the manifest, verifies the signature, asks the tracker where each chunk is, and downloads from peers in its rack first, its regional cache second, the origin last; the first machine in a region pulls from origin and seeds the cache and its peers.
4. Every chunk is hashed on arrival and rejected if it mismatches; when all chunks are present the file is assembled, the manifest hash re-verified, and the agent reports "complete, verified" and swaps a symlink the serving runtime reads.
5. The controller watches health gates (load success, latency, error rate, quality canary) and advances to 10, 50, 100 percent; rollback sets the desired version back, which is instant because the old files are still on disk.

5. Deep dives

How does 100 GB reach 10,000 machines in 20 minutes without crushing the network?

By turning fan-out into a tree and then a swarm. The origin sends one copy per region to the regional caches (a few copies, minutes). Within a region, agents in the same rack share chunks with each other: the first agent to get a chunk becomes a source for the others, so each chunk crosses the top-of-rack link about once and the rack's internal switch carries the rest. Chunks are requested in random order with rarest-first preference, the BitTorrent insight, so the swarm does not stampede the same chunk and every chunk is available from many peers quickly. Per-link rate limits in the agent keep cross-rack and cross-region usage under the 50 percent cap. The tracker is only a hint; an agent that cannot reach the tracker falls back to the regional cache, which falls back to origin, so distribution degrades to slower rather than to broken. This is a content distribution problem with the clients also acting as the edge.

Why content addressing, and what does it buy beyond integrity?

Naming chunks by their hash gives verification for free (the name is the checksum), makes any peer a valid source (a chunk is the same bytes wherever it comes from), and deduplicates across versions: two model versions that share most of their weights (a fine-tune, a changed head) share most chunks, so a 100 GB version may need only 5 GB of new transfer. The manifest signature protects the mapping from version to chunk hashes, which is what an attacker would need to alter to serve a tampered model; agents refuse any manifest whose signature does not verify against the publisher's key, and the serving runtime refuses to load a directory whose manifest hash is not recorded as verified. A corrupted chunk from a peer is detected by its hash and re-fetched from another source, with the bad peer noted.

How do we guarantee a machine never serves a partial or wrong version?

Atomic activation. Files are downloaded into a version directory that the runtime never looks at until a final step: after all chunks verify and the manifest hash matches, the agent writes a "verified" marker and flips a symlink from the current version to the new one in one rename, which is atomic on the filesystem. The runtime reloads from the symlink target at a safe point (between batches), and reports the version it is actually serving in its health endpoint; the rollout controller advances only on reports from the runtime, not from the agent, closing the loop. Disk pressure is handled by a retention rule (keep the serving version, the previous version, and the one being rolled out; delete the rest), enforced by the agent before it starts a download that would not fit.

How does this connect to the rest of the serving story?

The rollout gates are the serving system's: a canary cohort serves the new version to a slice of traffic while automated quality evaluations run on fixed prompts; the distribution system only promises that the bytes are right and present. A GPU fleet also needs the weights in accelerator memory, which takes a minute or two per machine; rolling out in waves keeps enough replicas serving while others reload, and the controller's percentage is really a cap on how many machines may be reloading at once.

6. What a strong answer sounds like

• Do the 1 PB arithmetic first and conclude that the origin cannot be the source for 10,000 machines; caches plus peer sharing follow.
• Chunk, hash and sign: content addressing gives integrity, any-source transfer and cross-version dedup in one move.
• Make activation atomic and gate the rollout on what the runtime reports, not on what was downloaded.
• Give the failure story: tracker, cache and origin each degrade to the next, never to a broken machine.
• Expected follow-ups: how to throttle during peak serving (distribution yields to traffic); how to pre-position a version to regions ahead of time; what the manifest signing key rotation looks like.
• Common mistake: a shared network file system mounted on every machine, which moves the fan-out problem to the file server and makes a slow mount a serving outage.

02Where this question was reported

• PracHub · Senior+, May 2026
• PracHub · PM Jun 2026: Gemini request path from prompt to grounded streamed answer.
• Prepfully · Candidate reports: LLM serving infra under low-latency, high-throughput.
• aie-blog.pages.dev · 'Distributed inference service for a 175B-parameter LLM'; TPU vs GPU, version integrity.