Microsoft and Hugging Face have published a benchmark that ignores what an agent says about its own work and grades the records it leaves behind instead.
ThinkingBox puts an agent through 507 stateful business workflows, each run 20 times from a fresh backend, and scores the terminal database state and the side effects of the run. The joint post describing it went up on 3 October 2026, alongside a paper on arXiv.
The headline number is uncomfortable for anyone shipping agents. In a common-set ablation covering 121,680 valid trials across 12 language models, 79,853 attempts failed the executable checks. Of those failures, 67.24% still terminated cleanly, invoked a state-changing tool and reported no final tool error.
Those are the vendor’s figures. ThinkingBox and ThinkingBox-Bench were built by the Microsoft Copilot Studio team in partnership with Toloka and academic collaborators, and no independent replication surfaced in searches on 11 October 2026.
The ticket that looked resolved and was not
The post opens with a customer story: a $745 kitchen appliance stuck in a courier exception at a Nashville distribution centre, fifteen days past its estimated delivery date. The agent works carefully. It pulls the order, checks tracking, looks up the profile, searches the refund policy twice, opens a ticket, documents the timeline and reads the policy correctly. Her account segment does not qualify for late-delivery compensation. It closes the ticket as resolved and asks whether there is anything else it can help with.
Two things are wrong. The carrier exception is still open, where the required end state was on hold pending resolution, and the customer never got an answer to what she actually asked.
That case is a benchmark task, and the customer is synthetic. The post describes it as adapted from sandbox_external_retail_group1.py:test_case_ST003_006, and the check it fails is a single field: the ticket status is solved where the required end state is hold. The benchmark’s disclaimer says every task in the public set is a synthetic reconstruction modelled on real enterprise patterns, and that the customers are not real.
What an executable check grades
Each task ships with a hidden, task-specific check. The paper’s Side-effect-centered judging section defines it as a conjunction over the terminal backend state, the extracted side effects and, on some tasks, the final response, with “conjunctive grading: a task passes only when all required conditions hold”.
Two consequences follow. An agent that reaches the correct end state by an unusual route still passes, because the check grades the world the agent leaves behind it and credits any trajectory that produces that state. An agent that writes the correct value while also touching the wrong record fails, because extra and missing effects count against it.
477 of the 507 tasks are graded on state alone. The other 30 add binary rubric questions for requirements that have no clean database value, such as whether the agent disclosed that a delivery date was not guaranteed.
The trust boundary is part of the design. The model sees the task, the dialogue and the tool schemas. The golden end state, the assertions, the grading internals and the credentials stay on the evaluator side.
How this differs from measuring code volume or activity telemetry
Two adjacent questions are easy to confuse with this one. Our earlier reporting on coding agents compared code volume against shipped software outcome, and our piece on Copilot agent activity looked at missing telemetry against actual use. ThinkingBox measures a third thing: the terminal state of a backend, and how often a model reproduces that state across repeated runs. A team with complete activity logs can still fail the field-level assertions this benchmark applies.
The failure percentages come from a 12-model ablation
The four headline shares all come from one slice of the campaign, which the post labels as a “common-set ablation covering 121,680 valid trials across 12 LLM models”. That is 507 tasks, 20 repeats each, 12 models. The same post ranks 18 models and shows 12 of them in its reliability figure, noting that the six it omits fall below 33% pass@1. It does not state which models the ablation used.
The denominator is valid trials, and the failure count is 79,853, or 65.63% of them. That 65.63% is not purely model behaviour. The post says system errors were counted as unsuccessful trials, and the OpenEnv adapter writes operational failures to an errors sidecar so they can be rerun and are never “silently mixed with model outcomes”. How many of the 79,853 were infrastructure errors is not published.
67.24% of failed runs still terminated cleanly
The post’s wording is that 67.24% of those failures “still terminated cleanly, invoked a state-changing tool, and reported no final tool error”. It defines clean termination no further, and the paper appendix that appears to hold the diagnostics behind these numbers was not retrievable when the research for this article was done. So clean termination is the post’s diagnostic wording.
The checks also recorded what went wrong inside those failures: wrong field values in 77.61%, unintended extra effects in 43.30% and missing required effects in 25.36%. Those findings overlap, so a single failed run can appear in more than one, and the three do not sum to a meaningful total.
Tool handling accounts for 79.9% of failures
This is a different measure with a different denominator, and keeping the two apart matters. Clean termination describes how a failed run looked from the outside. The failure signature classifies each failed trace under one dominant label, and the post’s summary is that “roughly four in five failures are tool handling, not reasoning”: tool usage 79.9%, wrong state updates 10.3%, incomplete user resolutions 7.0% and no state-changing action 2.9%.
The shares are unweighted averages of per-model shares drawn from observable labels, which the post describes as “not unique causal explanations”. The practical reading it offers is that agents often get far enough to attempt the workflow and then fail to recover from tool errors, failed preconditions or empty lookups.
Twenty runs per task separate breadth from consistency
Every task runs 20 times from an identical clean backend, and the benchmark reports three things: pass@1, the share of all attempts that succeeded; pass@20, the share of tasks solved at least once; and observed 20/20, the tasks that passed all 20 recorded attempts. The post is explicit that observed 20/20 is “the literal count” with “no estimator, no smoothing”, so it is not the same metric as a standard pass^k estimate.
The gap between breadth and consistency is the part worth carrying into a model decision. Kimi-K3 solves 476 of the 507 tasks at least once, or 93.89%, and passes 68 of them on all 20 attempts. Claude Opus 5 solves fewer tasks at least once, 401, and passes 241 on every attempt. Only three models keep most of their single-attempt score through 20 repeats: GPT-6 Astra retains 78%, and Claude Opus 5.5 and Claude Opus 5 each retain 71%.
Claude Opus 5.5 leads overall at 67.16%, two-thirds of a point above Claude Opus 5 at 66.50%. That lead comes from the post; in the paper revision read for this article, Claude Opus 5 is the top model at 66.50%. The post’s own conclusion is that the newer model does not improve dependability. Opus 5.5 passes exactly the same number of tasks on all 20 attempts, 241.
The figures diverge in two places we cite. The post’s cost table credits Qwen3.8-27B with 38 tasks passing 20 out of 20, or 7.50%, while the paper’s prose reports a pass^20 figure of 10.93% for the same model. The paper’s abstract reports Kimi-K3 at 17.60% pass^20, against the 68 of 507 tasks the post records for all 20 attempts, or 13.41%. Neither source explains the differences, which are likely literal counts against estimates, and this article does not pick one.
Cost forms part of the vendor’s argument, with limits worth stating in the same breath. The post prices each model’s recorded token usage at “undiscounted list rates available on OpenRouter+”, from a snapshot taken on 20 September 2026, and calls the result “a comparative efficiency index, not an invoice”. On that basis GPT-5.4 comes to $6.80 per dependable task across 128 dependable tasks, GPT-6 Astra $7.45 across 231, and Claude Opus 5.5 $7.80 across 241. Claude Opus 5 costs $13.30 for the same 241. These are list-rate estimates, and the snapshot has a date.
What the benchmark does not settle
The evaluation is vendor-run. Microsoft Copilot Studio built it with Toloka and collaborators from the University of Pittsburgh, Northwestern, Columbia and UC Irvine. Searches on 11 October 2026 found downstream relays of the post and no independent replication, which is an absence that one engine on one day cannot settle. One of those relays, FourWeekMBA, states directly that it verified nothing independently. The repositories are young as well, with 97 stars on the framework repo and 21 on the data repo at that check.
The tasks are synthetic, as the benchmark’s own disclaimer states.
The simulated user and the judge are language models. The paper audits 19,390 generated user follow-up turns with two LLM judges in a structured debate, with two people reviewing 120 sampled turns, and reports “strong corroboration of sampled Grounded labels but substantially lower confirmation of ungrounded labels”. Simulator and judge dependence appears among the paper’s own limitation headings, alongside the scope of the executable verdict and task construction and protocol.
What to check before you let an agent commit
Treat a repeat rate as a design input. Choose the k your workflow can justify, and say whether you are reporting best-of-k or every-of-k, because the two answer different questions.
Check the terminal state before committing, and do not rely on the model’s summary of it. This is the signal the benchmark grades on, and the post’s argument is that the same signal is available in production.
Classify tool and system errors so retries target the recoverable ones. Tool handling is the dominant failure signature at 79.9% of failures, averaged across the paper’s 14 non-RLFT models.
Cut the tool surface to what the workflow needs, which reduces the preconditions and lookups that can fail.
Require human approval on the changes you cannot cheaply reverse.
The post marks the whole list as untested: “We have not measured the lift from any of these on this benchmark, which is exactly the kind of thing the environment now makes testable.” The post’s evidence is the observed failure patterns; it measured no intervention.
Running one of the tasks yourself
All 507 tasks are readable in the Hugging Face dataset viewer at no cost. The executable source is the pinned release thinkingbox-bench-v1.0, released 17 August 2026 at commit fcaba4c, and the repository README notes that the viewer is a friendly representation while the GitHub repository “remains the executable source”. ThinkingBox code is MIT-licensed, the benchmark data is CDLA-Permissive-2.0, and the post says the OpenEnv environment ships under OpenEnv’s BSD-3-Clause licence.
Running one task end to end needs Linux or WSL, Python 3.11+, uv and Docker, a thinkingbox-data checkout at that pinned release, and model endpoints for the agent, the simulated user and the judge. A single endpoint can serve all three roles. Typesense 30.1 runs in Docker, the Session Proxy and the MCP servers start with tb mcp-start, and the OpenEnv server starts against a YAML config naming the three models. The readiness endpoint returns 503 until its data, configuration and Session Proxy checks pass, and it cannot check Typesense or probe every model endpoint, so those need confirming separately.
Scoring one episode uses the packaged evaluator, not the reset example that only resets and lists tools. Write - sandbox_external_retail_group1.py:test_case_ST002_001 into one_task.yaml and run thinkingbox-eval one_task.yaml --config "$PWD/thinkingbox.yaml" --repeat 1 --message-timeout 1800. The OpenEnv adapter is documented as designed for evaluation, returns a binary pass or fail reward, reports infrastructure failures separately so they are “never converted into benchmark failures”, and accepts a seed that it intentionally ignores because tasks are selected by UID.
We did not run any of this. Nothing in this article is a test of ours, and the figures remain the vendor’s.