Agent task success versus process quality
The natural way to evaluate an agent is to check whether it finished the task. It is necessary and it is not enough, because an agent can finish by a path you would never accept in production: by guessing, by trying everything, by doing something destructive and then undoing it, by calling a paid service forty times. This note separates the two questions, explains why success with a bad trajectory is common rather than rare, and sets out how to measure the trajectory without pretending there is only one right way to do a task.
Task success is an outcome metric: the end state matches what was asked. Process quality is a trajectory metric: the steps taken were efficient, safe, within policy and would work again. An agent can succeed with a terrible trajectory because the task did not happen to punish the bad steps this time. Outcome-only scoring cannot see that and therefore rewards it. Record the trajectory, score it on efficiency, unnecessary and repeated calls, policy violations, cost and recovery behaviour, compare it against a set of acceptable paths rather than one golden path, and read the two scores as a two-by-two: succeed with a good path, ship; succeed with a bad path, fix before scaling; fail with a good path, look at capability or environment; fail with a bad path, look at the agent.
Contents
01Two questions
- Task success asks about the end state. Was the ticket created with the right fields, does the test pass, does the answer match. It is checked once, at the end, usually mechanically.
- Process quality asks about the path. How many steps, which tools, in what order, with what side effects, at what cost, and how the agent behaved when something went wrong. It is checked over the whole trajectory, partly mechanically and partly by judgement.
The two are correlated in the way you would hope, good paths usually succeed, and uncorrelated in the way that matters: a success tells you almost nothing about the path that produced it.
02How an agent succeeds with a terrible trajectory
Each of these has shown up in real traces, and each would be scored as a pass by an outcome check.
- Trying everything. Asked to find the config value, the agent reads every file in the directory until one contains it. Success. Also two hundred tool calls, and a method that fails the moment the directory is large.
- Guessing an argument. The agent did not know the user's region, guessed "us-east", and the guess was right. Success, this time, on the one in three occasions it would be.
- Destroying and rebuilding. Asked to fix a failing test, the agent deletes the test, notices the suite passes, then writes a new test that passes. Success by the check; the original test's intent is gone.
- Spending. A research agent calls a paid search API forty times for a question that needed two queries. The answer is right. The cost is twenty times budget.
- Violating policy harmlessly. The agent ran a write before asking for confirmation, and the write happened to be what the user wanted. The rule exists for the times it is not.
- Going round in circles and getting out. Twelve steps of repeated identical calls, then on step thirteen a different approach that works. Success, after a trajectory that would have hit the step cap with a slightly different limit.
- Asking the user to do it. The agent asks the user six clarifying questions and effectively has them solve the task. The end state is correct.
The common thread is that the task did not punish the bad step on this occasion. The directory was small, the guess was right, the write was wanted. Evaluate on a hundred such tasks and the outcome score looks fine. Deploy to a hundred thousand sessions and every one of those bad steps finds the occasion that punishes it.
03Why outcome-only scoring rewards it
If success is the only signal, then every path to success is equally good, and the paths above are among them. Worse, when the outcome score is used to tune prompts or select models, it will favour whatever gets the end state most often, and "try everything" gets the end state very often. The evaluation is not neutral about trajectories; by ignoring them it actively selects for the robust-looking bad ones.
There is a second, quieter cost. Outcome-only scores give no diagnostic signal when they fall. A drop from 85 to 78 percent tells you something changed; it does not tell you whether the agent started picking wrong tools, stopped recovering from errors, or began running out of steps. The trajectory holds all of that.
04Process metrics worth recording
All of these come from the trace, which you should be storing anyway.
| Metric | What it catches | How to compute |
|---|---|---|
| Steps relative to reference | Trying everything; circling | Steps taken divided by the steps of a reasonable path; flag above 2× |
| Unnecessary calls | Reads that contributed nothing | Calls whose results are never referenced later, by a judge or by data flow |
| Repeated identical calls | State blindness | Count of calls matching an earlier call's name and arguments |
| Policy violations | Writes without confirmation, out-of-scope tools, destructive actions | Mechanical rules over the call sequence; any violation is a hard fail regardless of outcome |
| Side effects beyond the task | Destroy-and-rebuild; collateral changes | Diff of the environment's end state against the expected change |
| Cost and latency | Spending; slowness | Tokens, paid calls, wall time per task |
| Recovery behaviour | How it handled an error or an empty result | Inject a failure in a stub and judge the next steps: retried sensibly, changed approach, or repeated |
| Grounded arguments | Guessed values | For each argument, was its value present in the user's request or a prior result; unsourced values are flagged |
| Clarification quality | Asking too much or too little | Count of questions to the user against the task's ambiguity; both extremes score down |
| Reasoning quality | Sensible plan; sound inferences from results | A judge with a rubric over the trajectory; the softest metric, used for ranking not gating |
The grounded-arguments row deserves emphasis. A value the agent used that appears nowhere in its inputs is a guess, whether or not it was right, and it is detectable mechanically by searching the context for the value.
05Scoring against acceptable paths, not one golden path
The obvious way to score a trajectory is to compare it to a reference trajectory. The obvious problem is that most tasks have several reasonable paths: read the file then grep it, or grep then read; check the order first or the customer first. Scoring against one golden path penalises legitimate variation and rewards imitation.
Better options, from strict to loose:
- Required and forbidden calls. The path must include these calls, in any order, and must not include those. Catches the destructive detour and the skipped confirmation without caring about order.
- Ordered constraints. This call must precede that one: read before write, confirm before act. A partial order, not a total one.
- A set of acceptable paths. Enumerate the two or three reasonable sequences and score the closest match. Works when the task is small.
- Budgeted freedom. Any path is acceptable if it reaches the end state within n steps, within cost, with no violations. Lets the agent find paths you did not think of while still bounding waste.
- Judged quality. A rubric, applied by a model with human calibration, for the parts no rule captures: was the plan sensible, were the inferences sound.
In practice a task carries a few required calls, a couple of ordering constraints, a budget, and a rubric, and the process score is the combination. The required and forbidden calls and the policy rules are gates; the rest is a score.
06Reading the two scores together
| Good process | Bad process | |
|---|---|---|
| Succeeded | The case you want. Count it and move on. | The dangerous case. It passes outcome-only evaluation and will fail at scale. Fix before widening rollout; these cases are where prompt and tool-description work pays most. |
| Failed | The agent did the right things and still failed. Look at capability limits, the environment, a tool that returned wrong data, or a task that is underspecified. | The agent went wrong and the outcome shows it. The ordinary failure; the trajectory tells you where. |
The top-right cell is the reason to measure process at all. The bottom-left cell is the reason to measure it even for failures: it separates "the agent is not capable of this" from "the agent was unlucky", which are different roadmaps.
07In practice
- Store every trajectory, in evaluation and in production, with tool calls, arguments, results and the model's visible reasoning. Process metrics are derived from it after the fact; nothing here needs to be computed live.
- Make policy violations and forbidden calls gates, not scores. A success with a violation is a failure.
- Report success and process as two numbers, and plot the top-right cell's share over time. A rising share is an early warning that outcome scores are becoming hollow.
- When tuning, optimise on the combination. A prompt change that raises success by two points and doubles unnecessary calls is not an improvement.
- Use process metrics to pick what to look at. The trajectories with the worst process scores among the successes are the most informative traces to read by hand.
08Takeaways
- Success is about the end state; process is about the path. A success tells you almost nothing about the path.
- Bad trajectories succeed when the task does not punish them this time. At scale, every one finds the occasion that does.
- Outcome-only evaluation selects for robust-looking bad paths and gives no signal about why a score fell.
- Record steps, unnecessary and repeated calls, violations, side effects, cost, recovery and grounded arguments. All come from the trace.
- Score against constraints and budgets, not one golden path. Required and forbidden calls, ordering, a step budget, and a rubric.
- Read the two-by-two. Succeed-with-bad-process is the cell that matters most and the one outcome scoring cannot see.