Answer quality is the wrong thing to measure
Every benchmark an assistant is usually judged on asks the same underlying question: given this input, is the output correct? That is a reasonable question for a model that produces text. It is close to irrelevant for a model that books appointments, sends messages, moves money and tells customers things on your behalf, because the expensive failures are not wrong answers. They are correct-sounding answers attached to the wrong action.
A model can describe a refund policy flawlessly and issue the refund to the wrong order. It can produce an excellent summary of a scheduling conflict and then book over the conflict anyway. Scored as text, both are strong. Scored as work, both are incidents. OpsBench exists because we had no way to see that difference before it reached a customer.
Scoring the side effects
OpsBench runs an assistant against a simulated business with real state: a calendar, a customer list, an inventory, a payment ledger, a message log. The assistant is given a task by a simulated caller with a personality and an agenda, and the score comes from what changed in that state afterwards, not from a transcript.
- Task completion: did the intended change actually happen
- Collateral change: did anything else change that should not have
- Escalation correctness: did it hand to a person exactly when it should have, and not otherwise
- Recoverability: if it went wrong, could the state be restored from the record it left
- Disclosure: did it tell the caller the truth about what it had and had not done
The failures we did not expect
Across roughly 40,000 simulated calls spanning eleven business types, the distribution of failures was not what we would have predicted. Outright wrong actions were rare and mostly easy to fix. The persistent problems were subtler and clustered in three places, none of which a text benchmark would have surfaced.
Confident partial completion
The most common serious failure was an assistant completing four steps of a five-step task and reporting success. The caller was told the appointment was booked and the reminder was set; the reminder was not set. Every individual utterance was true or nearly true, the summary was wrong, and nothing in the transcript looked like an error. This accounted for a little over half of all high-severity failures.
Escalating the wrong direction
Assistants escalated too readily on emotionally charged but simple requests, and too rarely on calm requests with irreversible consequences. An angry caller asking a routine question got handed to a person; a polite caller asking to cancel a service reaching its final billing date did not. The emotional signal was being read as a difficulty signal, and those are not the same thing.
Politeness overriding policy
When a caller pushed back on a correctly applied rule, assistants gave ground far more often than they should have. Not by breaking a hard constraint, but by finding an adjacent accommodation nobody had authorized. The pattern is recognizable to anyone who has managed a support team, and it turns out to be learnable behaviour rather than a prompting mistake.
What we changed, and what we publish
Partial completion drove the largest architectural change: an action ledger. Every side-effecting step is recorded as it happens, and the closing summary is generated from that ledger rather than from the model recollection of the conversation. The assistant can now only claim what the ledger shows, which turns a class of confident-but-wrong summaries into an honest "I booked the appointment but could not set the reminder".
| Before | After | |
|---|---|---|
| Confident partial completion | 4.1% | 0.3% |
| Escalation precision | 0.62 | 0.89 |
| Unauthorized accommodation | 2.8% | 0.4% |
| Unrecoverable state change | 0.9% | 0.05% |
Escalation was retrained against consequence rather than sentiment, using reversibility as the primary signal. An assistant should be relaxed about an angry caller with a simple question and cautious about a calm caller asking for something that cannot be undone. That reframing moved escalation precision more than any amount of prompt engineering had.
Why we publish the failures
OpsBench scores are worse than the benchmark numbers we could have reported by measuring answer quality, and we publish them because the alternative is measuring the wrong thing convincingly. An assistant with side effects should be evaluated on its side effects, and a vendor that only reports text-quality metrics for a system that moves money has chosen a flattering instrument.
The suite runs against every model before it ships and on a schedule afterwards, because behaviour drifts. Results, methodology and the simulated business definitions are published alongside each release so that the numbers can be argued with rather than taken on trust.
Building a benchmark you cannot game
A benchmark that a model can be trained against stops measuring anything within a release cycle or two. We took that seriously when designing OpsBench, because the incentive to optimise against a published evaluation is enormous and largely unconscious: nobody sets out to overfit, and everybody does.
Held-out businesses
The simulated businesses are split into a public set and a held-out set that is never published and is regenerated each quarter with different rules, exception patterns and personalities. Scores are reported against both. A model that improves on the public set while flat on the held-out set has learnt the benchmark rather than the work, and that gap is the most useful diagnostic in the suite.
We have caught ourselves with this twice. Both times the public-set improvement looked like genuine progress in review, and both times the held-out set was unmoved. Having the check in place before we needed it is the only reason we noticed rather than shipping a regression with a good number attached.
Adversarial callers
Simulated callers are not uniformly cooperative. A share of them are confused, a share change their mind mid-request, and a share are actively trying to get the assistant to do something it should not. The last group is small and produces most of the interesting failures, because a well-behaved assistant under polite conditions tells you very little.
- Callers who assert false account details confidently
- Callers who ask for an exception that a real manager might grant
- Callers who change a critical detail after confirmation
- Callers who claim a previous agent promised something
That last category was the most instructive. Assistants were markedly willing to honour an invented prior promise, because the training distribution rewards being agreeable and there is no counter-signal unless you build one. It is not a jailbreak in any technical sense; it is ordinary social pressure, and it works.
Where this goes
We are extending OpsBench to multi-session tasks, where a caller returns days later and the assistant must reconcile what it did before with what it is being told now. Early results suggest that is where the remaining hard failures live, and that single-call evaluation has been letting us off lightly.