Existing agent benchmarks measure whether a model can browse a page, close a repository issue, or answer a hard question. Very few measure whether it can carry a working day: hold a plan across hours, read the documents that constrain it, act through tools bound to real accounts, and hand back something a person is willing to sign.
OpsBench is our attempt to measure that. It is built from anonymized workloads across professional services, logistics, healthcare administration, finance operations, and field work, and it scores side effects as well as answers.
Why a new benchmark
The work organizations need done is rarely one clever step. It is a chain: a request arrives, context is gathered from several systems that disagree, a rule set is applied, an artifact is produced, an action is taken, and someone is told about it. Every link is easy in isolation. Errors compound across the chain, and the cost of a small early mistake is a wrong document sent to a real person.
Most benchmarks also score only the final string. Operational work has consequences: a calendar moves, an invoice goes out, a record changes, a message leaves the building. A model can produce a perfectly reasonable-looking answer while having done the wrong thing on the way to it, and no answer-only score will notice.
Two properties follow from that, and both are things we wanted to measure directly. Work is long-horizon: the constraint that matters was established four steps ago. And work is consequential: the correct behavior is sometimes to stop and ask rather than to proceed confidently.
What it contains
OpsBench contains 1,840 tasks in eight categories. Each task ships with the source material, a sandboxed set of tools with real-looking state, and a rubric that covers both the artifact and the actions taken to produce it.
- Long-horizon planning. Overlapping constraints, lead times, dependencies between steps, and the one commitment that must not move.
- Rule-bound pricing. Line items derived from a published rule set, including edge cases the rules do not cover and which must be escalated rather than invented.
- Document reasoning. Contracts, statements, spec sheets, and scanned forms that contradict one another, where the task is to find the disagreement rather than average it.
- Reconciliation. Matching records across two systems that use different identifiers, where neither system is authoritative and the difference has to be explained.
- Tool use with consequences. Actions bound to live accounts, where the graded behavior includes knowing which actions require a person.
- Correspondence. Reading a thread, extracting what was actually promised by whom, and drafting the reply that closes it without inventing a commitment.
- Live conversation. Holding a position and re-planning in real time, on one surface among several, with no chance to revise before answering.
- Interruption and re-planning. The situation changes mid-task and everything downstream has to be re-derived rather than patched.
How we score it
Every task produces three independent scores, and a model has to earn all three. Reporting them separately is deliberate: a single blended number hides exactly the failure we care most about.
- Outcome accuracy. Is the artifact right: the correct numbers, the correct dates, the correct people?
- Side-effect correctness. Did the run change only what it was supposed to change? A task that produces the right answer by taking an unauthorized action scores zero here regardless of the artifact.
- Escalation quality. Where the task is deliberately underspecified, did the model ask instead of guessing, and did it ask the question that actually unblocks the work?
Escalation is scored in both directions. A model that refers everything upward is not safe, it is useless; a model that never refers anything upward is not capable, it is unsupervised.
Baselines
Results below are outcome accuracy on the public split. Side-effect and escalation scores are reported in the full results file, and the gap between the two columns widens on exactly the categories where the chain is longest.
| Category | Frontier baseline | Ruby-Horizon |
|---|---|---|
| Long-horizon planning | 71% | 93% |
| Rule-bound pricing | 78% | 96% |
| Document reasoning | 74% | 91% |
| Reconciliation | 63% | 88% |
| Tool use with consequences | 66% | 89% |
| Correspondence | 81% | 94% |
| Live conversation | 69% | 90% |
| Re-plan after interruption | 58% | 87% |
The category we find most diagnostic is reconciliation, where the frontier baseline scores lowest. It is the one task in the set that cannot be completed by producing confident prose: two systems either agree or they do not, and the model has to say which.
Availability
The numbers above are internal results from the first full runs of the suite. OpsBench is built from anonymized workloads, and no customer data is redistributed: every task is a reconstruction, with entities, figures, and identifiers regenerated so that no original record can be recovered from the set.
The suite is currently run in-house, alongside the controlled enterprise deployments the model is running in. We intend for it to be run by anyone building agents that do work for real organizations, and we are preparing the task split, tool sandbox, and scoring harness for that. A private split is held out to detect training against the benchmark; the held-out gap will be reported alongside any result we publish.