Design a CI system that runs a million tests in ten minutes

01The design, in the six-step framework

1. Requirements

Functional.

• A commit (or a pull request) triggers a run of every test that could be affected by it; a full run touches about one million tests.
• Results come back as a single pass/fail verdict per commit, with per-test logs attached.
• Flaky tests are detected and reported separately from genuine failures.
• Engineers can re-run a subset on demand.
• Out of scope: building the code itself (we assume a build service produces artifacts), code review tooling, deployment.

Non-functional.

• Wall clock: a full run of one million tests completes in under 10 minutes at the p95.
• Throughput: 2,000 commits per working day, peaking at 10 runs starting per minute.
• Correctness: a test result must be attributable to exactly one commit; a lost result counts as a failure, never as a pass.
• Cost: the fleet should not idle; under 15 percent of machine-hours wasted.
• Availability: 99.9 percent; a down CI blocks every engineer.

2. Core entities

• Run: run_id, commit_sha, trigger (push, PR, manual), state, started_at, verdict.
• Test target: target_id, path, owners, average_duration, resource class (CPU-only, needs a browser, needs a GPU), input hash.
• Shard: shard_id, run_id, list of target_ids, assigned worker, state, attempt number.
• Result: run_id, target_id, status (pass, fail, timeout, skipped, cached), duration, log_uri, attempt.
• Flake record: target_id, pass rate over the last N runs, last flip, quarantine flag.

3. API

• POST /runs {commit_sha, trigger, subset?} → {run_id}. Called by the source-control webhook or a person.
• GET /runs/{run_id} → {state, verdict, counts, shards}. Called by the PR page every few seconds, so it must be cheap.
• GET /runs/{run_id}/results?status=fail&cursor= → page of results with log links.
• POST /runs/{run_id}/retry {target_ids} → new attempt numbers for those targets.
• POST /workers/{id}/lease → {shard} and POST /shards/{id}/report {results}. Internal, between workers and the scheduler.

4. High-level design

One commit, end to end:

1. The webhook posts the commit. The API records a Run and hands it to the Planner.
2. The Planner walks the dependency graph from the changed files to the test targets that depend on them. For a typical commit that is 5,000 targets, not a million; only a change to a core library touches everything.
3. For each target it computes an input hash: the hash of the test's own sources, its dependencies' build outputs and its environment. If the result cache already has a pass for that hash, the result is copied in as cached and the target is not run.
4. The remaining targets are packed into shards of roughly equal predicted duration (about 30 seconds of work each) and pushed to the shard queue.
5. Workers lease shards, run them in sandboxes, stream logs to object storage and report results. A result write is idempotent on (run_id, target_id, attempt).
6. When the last shard reports, the Scheduler computes the verdict, applies the flake policy and marks the Run complete. The PR page sees it on its next poll.

5. Deep dives

How do we actually finish a million tests in ten minutes?

The simple answer is "add machines", and the arithmetic above says 17,000 cores at peak, which is affordable at Google scale but wasteful. Three things cut the real work. First, affected-target analysis: with a build graph, most commits touch under one percent of targets. Second, the content-addressed result cache: a test whose inputs have not changed since the last green run does not need to run again; on a monorepo this removes most of the rest. Third, duration-aware sharding: if shards are packed by count, one shard with three 90-second integration tests finishes last and the run waits for it. Packing by predicted duration (from the last 20 runs) with the longest tests first, a classic longest-processing-time heuristic, keeps the makespan close to the average. The remaining risk is the single slowest test; anything over two minutes is split into a shard of its own and started first.

What makes a cached result safe to reuse?

Only hermeticity. The input hash must cover everything the test can observe: its sources, every transitive dependency's compiled output, the container image, environment variables and declared data files. If a test reads an undeclared file, the hash misses it and a stale pass is reused. The usual defence is to run tests in a sandbox that only mounts declared inputs, so an undeclared read fails loudly instead of silently succeeding. A cached entry stores the result keyed by hash, with the run that produced it, and is invalidated only by a new hash; there is no time-based expiry because correctness does not decay with time.

How do we tell a flaky test from a real failure, honestly?

Retrying every failure once and reporting the retry hides real bugs. The policy that works: a failing target is re-run once on a different worker; if it passes, the result is recorded as flaky pass, not pass, and the Run's verdict is still green but the flake record for that target is updated. A target whose flake rate over its last 50 runs exceeds, say, 2 percent is quarantined: it still runs, its results are stored, but it no longer blocks verdicts and its owners get a ticket. Reporting the raw first-attempt result alongside the final one keeps the system honest and lets the flake dashboard show the true rate. The tempting shortcut, retrying up to three times and taking any pass, lets a 30 percent-flaky test pass 97 percent of the time and hides it forever.

What happens when a worker dies mid-shard?

A lease has a timeout, slightly longer than the shard's predicted duration plus a margin. The worker heartbeats every 10 seconds while running; if the Scheduler sees no heartbeat for 30 seconds it marks the lease expired and re-queues the shard with attempt+1. Results are written per target as they finish, keyed by attempt, so a partially completed shard does not repeat finished work twice and never double-counts. A shard that expires three times is marked failed with a reason of "infrastructure", which is reported separately from a test failure; a verdict is never silently green because results went missing. This is the same lease-and-heartbeat shape as any distributed job scheduler, and the same reasoning as failure detection anywhere else: distinguish slow from dead with a timeout you can justify.

6. What a strong answer sounds like

• Start with the arithmetic: 100,000 CPU-seconds per full run, so 167 cores per run for ten minutes, and say that the fleet size follows from concurrency, not from one run.
• Say that most of the win is not running tests: affected-target analysis and a content-addressed cache, and name the hermeticity condition that makes the cache safe.
• Explain sharding by predicted duration and why the slowest test bounds the makespan.
• Treat flakes as data: retry once, report the first attempt, quarantine by rate, and never let a lost result read as a pass.
• Expected follow-ups: how the dependency graph is built and kept accurate; how to prioritise PR runs over post-submit runs when the fleet is saturated (two queues, PRs first); how to handle tests that need special hardware (resource classes on shards).
• Common mistake: designing the fleet before asking which tests need to run at all.

02Where this question was reported

• PracHub · SWE, Sep 2026