The agent passes 92% of eval tasks and 60% of real sessions. What explains the gap?
Thirty points is not noise and it is not bad luck. It means the evaluation is measuring a different thing from the one users experience, and the two kinds of difference are worth separating: the sessions may be failing in ways the tasks never ask about, or "success" may simply mean something different in the two places. This note lists the structural differences between an evaluation task and a real session, gives a way to find out which ones are doing the damage, and then redesigns the evaluation so its number starts to mean what you thought it meant.
First check whether the two numbers measure the same thing: eval success is usually a grader confirming a task spec was met, and session success is usually a user signal, so part of the gap is definitional. Then sample a hundred failed sessions and categorise them. The usual causes are that real users are ambiguous where tasks are precise, give wrong details where fixtures are correct, change their minds and interrupt where tasks are single-turn, do not answer clarifying questions, arrive with intents the eval under-represents, and hit tool failures the stubs never produce. Redesign by building cases from real sessions with a simulated user, adding tasks where the right move is to ask or to notice an error, injecting tool failures, grading the whole session on the user's goal, weighting by the production intent mix, and tracking the correlation between offline and online scores as a metric in itself.
Contents
01First, are the two numbers measuring the same thing?
Before looking for agent failures, look at the two definitions of success.
- Eval success is typically: the grader confirmed that the task specification was met. The task said "cancel order 98765", the order is cancelled, pass.
- Session success is typically something else: the user said it helped, or did not escalate, or did not come back within a day, or completed a flow. Those are proxies for "the user's goal was met", and the user's goal is not the task spec.
These can diverge in both directions. A session where the agent correctly explained that the order could not be cancelled is a task success and often a user-signal failure. A session where the user gave up after three turns and did the thing themselves on the website is a user-signal failure that no task would represent. Estimate how much of the thirty points is this: take a sample of sessions marked failed, and ask whether the agent did what a task grader would have passed. If a third of them would pass, ten points of the gap are definitional, and the eval needs a different notion of success rather than harder tasks.
02Structural differences between a task and a session
| Dimension | Eval task | Real session | What breaks |
|---|---|---|---|
| Specification | Complete: the goal, the entity, the constraints are all stated | Ambiguous: "my order hasn't come", with three recent orders | The agent picks one, or asks badly, or does not ask |
| Correctness of inputs | The order id is right, the email is right | The user misremembers the id, gives the wrong email, describes the wrong product | The agent trusts the input and acts on the wrong thing, or finds nothing and gives up |
| Turns | One request, one resolution | Several turns; the goal changes mid-way; a second question is nested in the first | Context from turn one is applied to turn four; the second question is dropped |
| Cooperation | The task answers no questions, because there is no one to ask | The user ignores the clarifying question, answers a different one, or answers with a photo | The agent waits, or re-asks, or proceeds on a guess |
| Interruptions | None | "Actually never mind, can you instead…"; a long pause; a repeated message | The agent finishes the abandoned task anyway |
| Partial information | Everything needed is in the fixture | The record is incomplete; the tracking feed is stale; the policy has an exception not in the knowledge base | The agent answers confidently from incomplete data |
| Environment | Stubs return clean results | Tools time out, return errors, return malformed data | Untested error paths |
| Intent distribution | Even across the intents someone thought of | Heavy head, long tail, and intents nobody thought of | The eval over-weights what the agent is good at |
| User state | Neutral | Frustrated, in a hurry, hostile, or testing the bot | Tone failures; the agent takes bait |
| Length | Short | Long sessions with history that fills the context | State blindness; repeated calls; forgotten constraints |
| Judge of success | A grader with the spec | The user, with their goal | Definitional gap, as above |
An evaluation built from clean tasks tests the agent on the one row in each dimension that is easiest, and the agent passes. Users supply the other rows.
03Diagnosing which differences matter
Not every row costs the same, and which ones dominate depends on the product. Find out before redesigning.
- Sample a hundred failed sessions, stratified by intent. Read each with its trace and assign a primary cause from the table above, plus "agent error on a well-specified request" for the cases where the eval should have caught it, and "definitional" for the cases a task grader would have passed.
- Compare the intent distribution of the eval set to production. Compute what the eval's score would be if its cases were weighted by production frequency. A drop here means the eval is over-sampling what the agent does well.
- Check the clean-request subset. Among real sessions that were well-specified, single-turn and correct, what was the success rate? If it is near 92 percent, the agent is fine on the thing the eval measures and the gap is entirely structural. If it is 75, the eval is also too easy on its own terms, and that is a separate problem.
- Look at where sessions end. Abandonment after a clarifying question, abandonment after a wrong answer, escalation, and silent exit each point at a different row.
The output is a histogram of causes, and the redesign targets the tall bars.
04Redesigning the eval
- Build cases from real sessions. Anonymise failed and successful sessions and turn them into cases, with the user's side replayed by a simulated user. These carry the real distribution of ambiguity and error, which no one writes by hand.
- Add underspecified tasks where the right behaviour is to ask. "My order hasn't arrived" with three orders in the fixture. Pass requires a clarifying question that names the options, and then the right action after the simulated user answers. Score asking too much as well as too little.
- Add wrong-input tasks where the right behaviour is to notice. The user gives order O-98756 and owns O-98765. Pass requires detecting the mismatch, not acting on the typo and not giving up.
- Add mid-session changes. The user cancels the request on turn three and asks something else. Pass requires dropping the first task.
- Inject tool failures. In some fraction of cases the stub times out, returns an error, or returns a record with a missing field. Pass requires sensible recovery and honest reporting.
- Add long sessions. Ten or fifteen turns with a constraint stated early that must still be honoured late.
- Add the tail. The intents that appear in production below one percent each, which together are a quarter of sessions.
- Weight by production. Report the eval score weighted by the production intent mix, alongside the unweighted one.
- Add user state. Frustrated and hostile simulated users, and the ones who test the bot. Score tone and refusal to take bait.
The score will fall, perhaps a long way. That is the eval starting to agree with reality, and it is the point.
05Simulated users
Multi-turn cases need something to play the user. A scripted sequence of turns works for fixed flows and fails as soon as the agent asks an unexpected question. A simulated user is a model given a persona, a goal, the facts it knows, the facts it has wrong, and a behaviour profile, and asked to respond as that person would.
simulated user
goal get a refund for the damaged lamp
knows it arrived on Tuesday, the box was crushed
has wrong the order number (one digit off)
behaviour impatient; answers only the last question asked; mentions a second
unrelated question on turn three; gives up if asked for the order
number twice
stop when the agent confirms a refund or escalates, or after 12 turns
The simulated user is itself a model and needs its own checking: that it stays in character, does not reveal facts it was told it does not know, and does not helpfully fix the agent's mistakes. A small set of hand-reviewed transcripts per persona keeps it honest. Personas should be drawn from the diagnosis in section 03, so the eval's population of users resembles the real one.
06Grading a session, not a task
With multi-turn cases, the grader reads the whole transcript and answers a different question: was the user's goal met, as the user would judge it? That includes:
- Did the agent resolve the goal, or correctly explain why it could not and offer the next step?
- Did it handle the wrong input, the interruption, the second question?
- Did it ask when it should and not when it should not?
- Did the user, as simulated, reach a state they would call done, or did they abandon?
- Process quality over the session, as in the success-versus-process note: violations, waste, guessed arguments.
Abandonment is a failure, even when the agent's last message was correct. The user's leaving is the signal; the grader should treat a simulated user who gave up as the real one would be counted.
07Evaluating the eval
An evaluation's purpose is to predict production. So measure that directly: across releases, plot the offline score against the online success rate, and track the correlation. If a change raises the offline score and online does not move, the eval is measuring something users do not care about, and that is a bug in the eval. Over time the redesigned eval should move the two numbers together, and the gap between them should settle to a stable offset that you understand, the definitional part, rather than a thirty-point mystery.
Keep feeding it. Every week, the failed sessions that the eval would have passed are the cases to add. The eval is never finished; it is a model of your users, and your users change.
08Takeaways
- Check the definitions first. Task-spec success and user-goal success differ, and some of the gap is that difference.
- Real sessions are ambiguous, wrong, multi-turn, uncooperative, interrupted, partially informed, long and unevenly distributed. Clean tasks test none of that.
- Diagnose before redesigning. A hundred failed sessions, categorised, and a production-weighted rescore.
- Build cases from real sessions with simulated users, and add tasks where the right move is to ask, to notice an error, to drop an abandoned goal, or to recover from a tool failure.
- Grade the session on the user's goal, count abandonment as failure, and score process as well as outcome.
- Track offline-to-online correlation as a metric. An eval that does not predict production is the thing that is broken.