An LLM regression suite in CI: size, runtime, thresholds and flakes

01Two tiers, and why one is not enough

One tier cannot do both jobs. A suite small enough to run on every change is too small to detect a three-point drop with any confidence, because the confidence interval on a hundred cases is about ten points. A suite large enough to detect that drop takes too long and costs too much to run on every change. So the smoke set answers "did this change break something obvious" quickly, and the nightly set answers "is the system as good as yesterday" thoroughly.

The smoke set is chosen, not sampled. It holds one or two representative cases per intent or feature, every safety and policy gate, the cases that have regressed before, and the cases that exercise each tool. The nightly set is everything.

02What a case is

A case fixes everything except the model's behaviour, so that a different result means the system changed.

case support-return-0142
  input         the user's message(s); for multi-turn, the scripted turns or a simulated user
  fixtures      the state the stubbed tools return: this order, this customer, this policy
  clock         a fixed "now", so relative dates resolve the same way every run
  expected      the outcome: end state, required tool calls, forbidden calls, answer facts
  grader        which checks apply and how they combine
  tags          intent, feature, source (production failure, synthetic, red team), gating or not

Stubbed tools matter more than anything else here. A case that calls a live service drifts as the service's data changes, and cannot be made to return an error on demand. Stubs make each case deterministic on the environment side and let you test error handling by returning errors.

03Graders, from cheap to expensive

• Mechanical checks. The response parses; the required tool was called with the critical arguments correct, compared by schema type; no forbidden call; no policy violation; the end state matches. Free, deterministic, and most of the smoke set.
• Reference comparison. Answer facts present, normalised string or set comparison. Cheap.
• Model judge with a rubric. Correctness, completeness, faithfulness to tool results, tone. Costs a model call per case and is itself non-deterministic, so the judge's prompt and model are versioned, and the judge is calibrated against a set of human-labelled responses before it is trusted. Changing the judge is a change to the suite and resets the baseline.
• Human review. A sample of the nightly run, weighted toward cases where the judge was uncertain or disagreed with the mechanical checks. Keeps the judge honest.

A case's grade combines these, with mechanical checks as gates and judged dimensions as scores.

04Thresholds that respect noise

A fixed bar, "95 percent must pass", has two problems: the right bar differs by dimension, and a run that scores 94 may be noise rather than regression. Better:

• Gates at 100 percent for safety, policy and forbidden calls. One failure blocks. These are the cases where a single miss is a real incident.
• Per-dimension bars for the rest: tool selection, argument accuracy, answer correctness, format. Each has its own current level and its own tolerance.
• Compare to the baseline, not to an absolute. The baseline is the last accepted run on the main branch. A change is a regression if a dimension falls by more than the noise band, which you measure by running the baseline several times and recording the spread. Two runs of the same system typically differ by one to three points on a few hundred judged cases; a drop inside that band is not evidence.
• Case-level diffs over aggregate scores. The useful signal is "these twelve cases passed yesterday and fail today", with their traces. An aggregate that moved half a point while thirty cases flipped in each direction is hiding something.
• Allow a deliberate drop with a sign-off. Sometimes a change trades two points of one dimension for five of another. The suite should make that visible and require a human to accept it, not block it forever.

05Flakes: non-determinism is not regression

A model at temperature zero still gives different answers run to run, and a judge is a model too. A suite that treats every failure as a regression will block merges on noise and be ignored within a month. The policy:

• Run each case more than once. Three runs; a case passes if all three pass, or if two of three pass and the case is not a gate. Record the per-case pass rate over time; a case at 60 percent is telling you something about the system's stability on that input, which is itself a finding.
• Re-run failures before reporting. A case that failed once and passes twice on re-run is a flake, logged as such. A case that fails on re-run is a regression.
• Cache model responses by request hash. Hash the full request, including model version, tools, system prompt and messages. If the hash is unchanged from the baseline run, reuse the cached response. This makes the unchanged parts of a run free and deterministic, and means a pull request that touches one prompt only pays for the cases that prompt affects.
• Pin model versions. Use dated or versioned model identifiers, never aliases that move. A version change is a deliberate event that triggers a full nightly run and a new baseline.
• Quarantine inconsistent cases. A case whose pass rate hovers around 50 percent across many runs is not a useful gate. Move it to a non-blocking list, investigate why the system is unstable on it, and either fix the system or rewrite the case.
• Keep the judge's temperature at zero and its prompt fixed, and measure its own agreement with itself on repeated runs. A judge that disagrees with itself ten percent of the time adds ten points of noise to everything it grades.

06Runtime and cost budget

• Parallelise. Cases are independent; run them concurrently up to the rate limit. A hundred cases at ten seconds each is seventeen minutes serial and under two minutes at ten wide.
• Stubs, not services. Removes network time from every tool call and makes the only latency the model's.
• Cache, as above. On a typical pull request most requests are unchanged.
• Cheaper judges for the smoke tier. A small model judging a narrow rubric is fast and good enough to catch obvious breakage; the full judge runs nightly.
• Track cost per run as a metric in its own right. A suite whose cost climbs unnoticed gets cut, usually at the wrong moment.

07What triggers which tier

08Data hygiene

• Feed production failures into the suite. Every incident becomes a case, anonymised. The suite's coverage then follows where the system actually breaks.
• Hold out a set you never tune on. If prompts are edited until the suite passes, the suite stops measuring generalisation. Keep a held-out slice that is run nightly and never looked at case by case.
• Watch for leakage. Evaluation inputs that appear in few-shot examples or in a retrieval index inflate scores. Check overlap when either changes.
• Deduplicate and rebalance. A suite that grows by appending production failures over-represents whatever broke last month. Rebalance by intent against the production distribution periodically.
• Version the suite with the code. Cases, fixtures, graders and baselines live in the repository, so a result is reproducible against the commit that produced it.

09The report

The output a reviewer sees on a pull request should fit on one screen: each dimension's score against the baseline with the noise band; the gate results; the list of cases that flipped, pass to fail and fail to pass, each linking to its trace; the flakes re-run and resolved; and the run's cost and duration. The aggregate score is the least useful line on it.

10Takeaways

1. Two tiers. A chosen smoke set on every change for obvious breakage; a large nightly set for drift and the tail.
2. A case fixes inputs, tool state, clock and grader. Stubs, not live services.
3. Gates at 100 percent for safety and policy; per-dimension bars against a baseline with a measured noise band for the rest.
4. Three runs per case, re-run failures, cache by request hash, pin model versions, quarantine the inconsistent.
5. Case-level flips with traces are the signal; the aggregate is the summary.
6. Production failures become cases; a held-out slice is never tuned on.