Evaluating safety behaviour: harmful compliance and over-refusal together

01Safety is a classifier with two error types

Strip away the content and a safety-tuned model is making a binary decision on every request: help, or decline. Against the truth of whether the request should be helped, that gives four outcomes.

Any change that moves the decision boundary moves both error rates in opposite directions. A stricter system prompt, a more cautious model, a keyword filter: each reduces harmful compliance and increases over-refusal. There is no free move, only a choice of where on the trade-off to sit, and that choice cannot be made with one of the two rates hidden.

02Why optimising one rate produces a useless product

Harmful compliance on its own has a trivial optimum: refuse everything, and the rate is zero. Nobody sets out to do that, but every incremental step toward it looks like progress on the dashboard, and nothing on the dashboard pushes back. The result is a model that declines to explain how to kill a Python process, will not discuss the maximum safe dose of paracetamol, refuses to help a security engineer understand an exploit they are patching, and lectures a novelist about violence in fiction.

That model is not safe; it is useless in a way that has its own harms. Users route around it, to models with less care or to search engines with none. Legitimate professional uses, medicine, security, law, harm reduction, are the ones most likely to be refused, because they are the ones that sound closest to the harmful set. And the organisation gains a false confidence, because the one number it watches is excellent.

The over-refusal rate is the number that pushes back. Measured alongside harmful compliance, it turns "we made the model safer" into "we moved the operating point", which is a decision someone can actually make.

03The harmful set

Requests the model should decline, or decline the harmful part of, organised so the results can be read by category and severity.

• Categories matching your policy: violence, weapons, self-harm, sexual content involving minors, fraud and financial crime, malware and intrusion, privacy violations, hate, dangerous misinformation, and whatever your domain adds. Each category is reported separately; an aggregate hides which categories leak.
• Severity tiers within each category. A request for a general overview is not the same as a request for operational detail, and the acceptable compliance rate differs: near zero for the top tier, and for the lowest tier a safe partial answer may be the right behaviour rather than a refusal.
• Adversarial variants of each base request, because the plain phrasing is the one that is easiest to refuse and least representative of attempts. Role-play framing, hypothetical and fictional framing, requests split across turns, encoded or obfuscated text, appeals to authority or research, and the "for safety reasons I need to know" inversion. Each base request should exist in five to ten variants, and the per-variant compliance rate shows which framings get through.
• Refresh. New framings appear constantly. A quarterly red-team exercise adds variants; a set that is not refreshed measures last year's attacks.

04The benign set, as matched pairs

Requests the model should help with, chosen so that they are as close as possible to the harmful set in surface form. A benign set of ordinary questions about cooking and travel measures nothing; the model was never going to refuse those. The set has to live on the boundary.

The strongest construction is matched pairs: for each harmful request, a legitimate request on the same topic, using similar vocabulary, that a reasonable person would expect help with.

Add a tier of ordinary-but-sensitive requests that are not paired, medical, legal and financial questions from the general public, so that the over-refusal rate also covers the everyday case. And add requests that mention a sensitive topic without asking anything about it, "I work in oncology and need help with this spreadsheet", because keyword-driven refusal fails there in a particularly irritating way.

05Labelling responses

Binary helped-or-refused labels miss the best behaviour, which is often to help with the legitimate part and decline the harmful part. Use three response labels and a separate harm grade.

• Full compliance. The model did what was asked.
• Safe completion. The model helped with the legitimate need, declined or redirected the harmful element, and said so. On a self-harm request, that is helplines and a conversation rather than methods. On a dual-use security question, that is the conceptual explanation without the operational payload.
• Refusal. The model declined, with or without explanation. Note whether it explained and whether it offered an alternative; a bare refusal is worse for the user than a redirect.
• Harm grade, on the content actually produced, independent of the label. A "refusal" that includes the harmful detail before refusing scores as harmful. A "full compliance" on a benign request scores as harmless. The label is about the decision; the grade is about what came out.

Grading is done by a model judge with a per-category rubric, calibrated against human labels on a few hundred responses, with disagreement cases reviewed by people. The judge's own over- and under-sensitivity is measured on the calibration set and reported; a judge that calls safe completions "compliance" will make the model look worse than it is.

06Metrics and the operating point

harmful compliance rate   on the harmful set:   responses labelled full compliance, or graded harmful
over-refusal rate         on the benign set:    responses labelled refusal, plus safe completions that withheld the legitimate answer

report both, per category, per severity tier, per adversarial framing
never report either alone

Plotting the two against each other across configurations, different system prompts, different models, different filter thresholds, gives a trade-off curve, and the product decision is where on the curve to sit. The operating point should differ by category: for the highest-severity categories accept more over-refusal to drive compliance near zero; for medical and security questions from professionals accept a little more compliance risk to keep the product usable. Those are policy decisions, and the evaluation's job is to make them explicit rather than to make them.

A single blended "safety score" is tempting for a dashboard and should be resisted; it re-hides exactly the trade-off the two numbers were separated to show.

07Multi-turn and agentic contexts

• Multi-turn. Many real attempts build up over turns, each innocuous. The set needs scripted conversations, not only single requests, and the grade applies to the conversation.
• Tool use. In an agent, refusal looks different: declining to run a legitimate action, or running a harmful one. Over-refusal in an agent is a support bot that will not cancel an order because the request mentioned "fraud". Harmful compliance is a tool call that should have been declined. Both need cases in the agent's own evaluation, using the same two-sided structure.
• Injected instructions. Content arriving through tools, documents and pages that tries to redirect the model is a safety input too, and the benign side exists here as well: a document that discusses a sensitive topic is not an attack.

08Regression rules

Both sets run in the nightly suite from the regression note, with the gating cases, the top severity tier, in the smoke set.

• Top-tier harmful cases are gates at zero compliance.
• Over-refusal has a budget per category, set from the current operating point. A change that lowers harmful compliance and raises over-refusal past its budget fails the suite, exactly as a change in the other direction would.
• Report the pair on every change. A reviewer should see both numbers move, and should be asked to accept a trade, not to celebrate one side of it.
• Feed production into both sets. User complaints about refusals are the over-refusal set's best source; flagged outputs are the harmful set's.

09Takeaways

1. Two error types. Helping when it should refuse; refusing when it should help. Every change trades one for the other.
2. One rate has a trivial optimum: refuse everything. Measured alone, it leads there.
3. Build the benign set as matched pairs on the harmful set's topics, plus everyday sensitive questions and sensitive-sounding idioms.
4. Three response labels and a separate harm grade. Safe completion is often the right answer, and a refusal can still leak.
5. Report the pair, per category and severity, and choose the operating point per category. No blended score.
6. In regression, each rate has a budget, and a change may not spend one to buy the other without a person agreeing.