AdaPlanBench vs ToolMaze vs TASTE vs K-BrowseComp: Agent Stress Tests
Four 2026 benchmarks stress-test agents: AdaPlanBench's hidden constraints (67.75% best), ToolMaze's silent tool failures (−37% PRR).
The question all four benchmarks ask
Static agent benchmarks keep saturating, so the field’s measurement problem has moved: it is no longer “can the agent do the task at all” but “what happens when the task stops cooperating.” Four benchmarks published within two weeks of each other in May–June 2026 attack that question from four different directions, and a companion analysis paper explains the mechanism underneath. Together they give a more honest picture of agent robustness than any single leaderboard number.
The four stress axes are: constraints revealed mid-task (AdaPlanBench), tools that fail while the agent is using them (ToolMaze), tasks synthesized to be harder than the training distribution (TASTE and its τᶜ-Bench), and web questions grounded in a non-English language (K-BrowseComp). The mechanism paper asks what happens to a search agent’s context as it grows, and when pruning stale observations helps or hurts (masking stale observations).
Each benchmark isolates one failure mode that demos hide: a polished demo shows the agent succeeding on the first try with working tools, English questions, and a finished plan. Real deployments offer none of those guarantees.
AdaPlanBench: stress from constraints that accumulate
AdaPlanBench (COLM 2026) starts from household planning tasks and makes them adversarial in a specific way. Each of 307 tasks is augmented with dual constraints: world constraints (what the environment allows) and user constraints (what the user actually wants). The agent does not see these constraints up front. It proposes a plan, receives feedback only when the plan violates a hidden constraint, and must revise iteratively as the feedback accumulates.
That protocol matters because it mimics how planning actually fails in products: requirements surface during execution, not before it. The authors evaluate 10 leading LLMs on this loop, and the best model reaches 67.75% accuracy. The headline finding is directional rather than scalar: performance degrades as constraints accumulate, user constraints are the harder of the two, and failures cluster around weaker physical grounding: models that can recite planning theory but lose track of what a household environment actually permits.
The number to quote is the 67.75% ceiling. Two-thirds accuracy on household tasks, with only ten models tested and constraint feedback handed to the agent for free, is a long way from “agents can plan.” If your deployment involves users who change their minds, read this benchmark as the closest proxy.
ToolMaze: stress from tools that fail
ToolMaze inverts the usual assumption. Every agent benchmark gives the agent working tools; ToolMaze gives the agent a 2 × 2 taxonomy of tool perturbations and measures what happens when tools fail mid-task, across tasks of varying DAG complexity. The metric is Perturbation Recovery Rate (PRR): after a tool fails, does the agent notice, diagnose, and replan?
Two findings carry the paper. First, the sharpest drops come from implicit semantic failures: tools that return wrong-but-plausible output rather than errors. When a tool fails loudly, agents recover; when it fails silently, PRR drops by roughly 37%. Second, scale does not rescue recovery: the authors report that agentic fault-tolerance improves with model scale about 3.66× more slowly than basic execution ability. Bigger models get better at the happy path faster than they get better at the failure path, which means the robustness gap widens with capability, not narrows.
If your agent calls external APIs, scrapers, or user-provided tools (anything that can return stale, partial, or wrong data without raising an exception), ToolMaze’s distinction between loud and silent failure is the most actionable single idea in this group.
TASTE and τᶜ-Bench: stress from tasks harder than your benchmark
TASTE attacks a different problem: agent benchmarks are static, so models overfit to them. Instead of hand-writing harder tasks, the authors generate them: sample tool sequences, cluster them, instantiate candidate tasks, and run an iterative difficulty evolution loop driven by an Adaptive Contrastive n-gram model with LLM-judged validity signals. The output is τᶜ-Bench, built across three τ²-Bench domains, and the synthesized tasks more than double the number of unique tool combinations the underlying domains covered.
The evaluative punchline uses 11 agent/user LLM pairs: on the evolved tasks, Gemini-3-Flash pass rates fall from 0.82–0.94 down to 0.28–0.61. A model that clears nine of ten standard tasks can clear fewer than one in three once the task distribution moves, even though the domain and tools are nominally the same. This is the benchmark-gaming answer: if you suspect a vendor number was tuned to visible tasks, a synthesis pipeline like TASTE’s is how you regenerate pressure the model has not already absorbed.
Note what TASTE is not: it does not claim evolved tasks are more “realistic.” It claims they are more covered-resistant, and the 0.82-to-0.28 collapse quantifies how much apparent competence was sitting in task familiarity.
K-BrowseComp: stress from non-English ground truth
K-BrowseComp grounds web-browsing agents in Korean contexts. It contains 400 problems: a 300-item verified split built and checked by native Korean speakers, plus a 100-item synthetic split generated with hard few-shot examples and failure-targeted generation, released separately as a stress test.
The results cut against two comfortable assumptions at once. Frontier models (the paper names GPT-5.5, DeepSeek-V4-Pro, and GLM-5.1) score far lower than they do on English BrowseComp, landing at 30.00–45.67% on the verified split. And the Korean LLMs evaluated from the national foundation-model program score 0.00–10.33%. The best frontier model on the adversarially filtered synthetic split manages 26.00%. Data and code are public.
The benchmarking lesson generalizes beyond Korean: agentic capability measured in English does not transfer cleanly to other languages, and “web access” is not a language-neutral skill: retrieval, page understanding, and answer verification all degrade when the ground truth moves out of English. For any team deploying agents outside English-first markets, this is the cheapest way to find out how much of the English number was language luck.
Key numbers
| Benchmark | Stress axis | Scale | Headline numbers |
|---|---|---|---|
| AdaPlanBench | Hidden world + user constraints revealed through iterative feedback | 307 household tasks, 10 LLMs | Best model 67.75%; performance degrades as constraints accumulate; user constraints hardest |
| ToolMaze | Tool failures, loud and silent, across DAG complexity | 2 × 2 perturbation taxonomy | PRR drops ~37% under implicit semantic failures; fault-tolerance scales 3.66× slower than execution |
| TASTE / τᶜ-Bench | Synthesized tasks harder than the original benchmark | 3 τ²-Bench domains, 11 agent/user LLM pairs | Gemini-3-Flash pass rate 0.82–0.94 → 0.28–0.61; >2× unique tool combinations |
| K-BrowseComp | Korean-language web browsing | 400 problems (300 verified + 100 synthetic) | Frontier 30.00–45.67% on verified; Korean LLMs 0.00–10.33%; best on synthetic split 26.00% |
Read this table as four one-dimensional cuts, not a ranking. No model appears across all four, no harness is shared, and the metrics (accuracy, recovery rate, pass rate, answer accuracy) are not convertible into each other. What is comparable is the shape of the finding: in every case, moving one environmental variable breaks agents far more than model-card numbers suggest.
Why stale-context masking is the mechanism paper
The masking study explains part of why agents degrade under stress at all. As a long-horizon search agent accumulates retrieved observations, its context fills with information that goes stale. The authors test masking stale observations across backbones from 4B to 284B parameters and three retrievers, and find an asymmetric inverted-U: masking helps little with a weak retriever, peaks when a strong retriever meets a mid-capacity model, and collapses when the model is saturated; past that point, masking hurts.
The mechanism is retriever recall interacting with the model’s implicit filtering, and the authors frame masking as a token-for-turn trade-off: you buy turns of fresh context at the cost of tokens that might have mattered. The connection to the four benchmarks is direct. AdaPlanBench’s agents drown in accumulating constraint feedback, ToolMaze’s agents must keep failed-tool state salient, and K-BrowseComp’s agents juggle multilingual pages: each is, among other things, a context-management problem, and this paper is the one that maps where context pruning stops being free.
What the four benchmarks agree on
Three claims repeat across all four papers. First, execution is not recovery: agents score far higher on the happy path than on any perturbation of it: 67.75% with free constraint feedback, 0.82-plus pass rates collapsing to 0.28 on evolved tasks, and frontier English-browsing scores dropping to 30–45% in Korean. Second, the failures are silent: the worst drops come from implicit semantic failures and unfamiliar-but-plausible tasks, not from loud errors agents can route around. Third, scale is not robustness: ToolMaze’s 3.66× scaling gap says capability grows faster than fault-tolerance, so bigger models can be simultaneously more impressive and more brittle.
When to use which
- Your users change requirements mid-task, or the environment enforces hidden rules: AdaPlanBench’s constraint-reveal protocol is the closest off-the-shelf design, and its 67.75% ceiling is the calibration point.
- Your agent depends on external tools that can fail silently: ToolMaze’s loud-versus-silent distinction is the design insight; its ~37% PRR drop under implicit failures is the number to beat.
- You suspect benchmark overfitting, from a vendor or your own training loop: TASTE’s synthesis pipeline shows how to regenerate task pressure, and the 0.82–0.94 to 0.28–0.61 collapse is the warning shot.
- You deploy outside English: K-BrowseComp’s 30.00–45.67% versus 0.00–10.33% split quantifies how much of an English agent number transfers, usually less than half.
- Your agent runs long horizons with growing retrieval context: run the masking ablation before shipping any pruning strategy; the inverted-U says the answer depends jointly on retriever strength and model capacity, not on either alone.
Limits and open questions
All five papers are recent (May–June 2026), and the model generations they evaluate will age quickly; the protocols are the durable part. The four benchmarks share no models, no harnesses, and no metrics, so no same-harness comparison exists between them, and any composite “agent robustness score” across the four would be manufactured rather than measured. AdaPlanBench’s household domain and ToolMaze’s DAG tasks are synthetic environments whose transfer to open-ended work is unproven; TASTE’s evolved tasks trade realism for coverage resistance, so a low τᶜ-Bench score does not by itself mean a model is bad at the underlying domain; K-BrowseComp covers one language, and its synthetic split’s 26.00% ceiling partly reflects adversarial filtering rather than pure difficulty. The masking study is an analysis, not a benchmark, and its 4B–284B sweep does not guarantee the inverted-U holds at larger scales or with newer retrieval pipelines. Finally, all four benchmarks can in principle be gamed once their task pools are absorbed into training data; TASTE’s pipeline is the one answer the group offers to that problem.
FAQ
Which agent stress-test benchmark should I start with?
Match the benchmark to your dominant failure mode, not to which number looks most impressive. Constraints that surface mid-task point to AdaPlanBench; unreliable tools point to ToolMaze; suspected benchmark overfitting points to TASTE; non-English deployment points to K-BrowseComp. If you genuinely do not know your failure mode, ToolMaze is the cheapest starting point because silent tool failure is the most common gap between demos and production.
Can I compare AdaPlanBench, ToolMaze, and K-BrowseComp scores directly?
No. They use different models, harnesses, task scales, and metrics: 67.75% accuracy, a ~37% PRR drop, and 30.00–45.67% answer accuracy are not convertible into one robustness score. Compare the shape of findings (agents execute better than they recover, everywhere) rather than the numbers.
What is the single most surprising number in this group?
Two candidates. TASTE’s Gemini-3-Flash collapse from 0.82–0.94 to 0.28–0.61 shows how much apparent agent competence can live in task familiarity rather than capability. And ToolMaze’s 3.66× gap shows fault-tolerance scaling more slowly than execution, meaning the robustness deficit grows with model size, not shrinks.
Do these benchmarks prove current agents are unusable?
No. They prove agents are uneven: strong on first-pass execution, weak on recovery, perturbation handling, distribution shift, and non-English ground truth. AdaPlanBench’s best model still clears two-thirds of constrained tasks; K-BrowseComp’s frontier models still answer roughly a third to a half of Korean browsing questions. The unevenness, not the average, is what to engineer against.
Where does agent evaluation go after these four?
Toward synthesis and rotation: TASTE’s pipeline is the clearest template for regenerating task pressure faster than models can absorb it, and the masking study adds the context-management dimension the four benchmarks all brush against. Expect the next generation of benchmarks to combine stress axes (constraint revelation plus tool failure plus non-English retrieval) in single environments.