DRIFT vs Arbor: Diagnosing vs Preventing Deep-Research Agent Failures
DRIFT finds the span where a deep-research trajectory breaks (54.91% macro-F1, up to 30 points over raw reads); Arbor blocks bad branches with hypothesis trees and a held-out gate (86.36% Any Medal on MLE-Bench Lite).
The question both papers are circling
A deep-research agent can burn through a dozen search-read-reason cycles and still return a wrong answer. The final score says the run failed; it says nothing about where. Two 2026 papers start from that same blind spot and walk in opposite directions. DRIFT looks backward: given a finished trajectory, point to the exact span where an unsupported claim became load-bearing for the wrong answer. Arbor looks inward: while the work is still running, keep every attempt in a hypothesis tree and let a branch merge only if it survives held-out evaluation. One diagnoses failures after the fact; the other builds a process that refuses to admit many failures in the first place.
Outcome-only scoring is the shared enemy. A capability benchmark such as Agents’ Last Exam records whether a long-horizon run succeeded, which is useful, but neither paper treats that as a debugging signal. Their bet is the same: agent reliability is limited by state management, and you fix it by making more of the run inspectable.
DRIFT: post-hoc diagnosis with claim auditing
The DRIFT paper reframes debugging as span-level error localization. A trajectory is cut into semantic spans (11.95 on average, roughly a dozen per run), and the task is to flag the spans where a harmful error lives while leaving healthy exploration alone. “Harmful” is the load-bearing word: a dead-end query the agent recovers from is not an error, and a detector that flags normal exploration is useless.
The benchmark built for this, TELBench, is the paper’s most durable artifact. The authors collected 2,790 raw trajectories from 2 agent frameworks (MiroFlow and OAgent) running 3 backbone models (GPT-5, Gemini-2.5-Pro, Claude-Sonnet-4.5) on 3 benchmarks (GAIA-val, XBench, and BrowseComp-test, 200 tasks each, about 465 tasks in total). After normalization and expert review with LLM assistance, 1,000 verified instances remain, split 600 easy and 400 hard, each labeled with its harmful spans. The annotation schema catalogs 18 primary faults in 6 families: constraint handling, search and retrieval, evidence grounding, entity mapping, information processing, and process control. That vocabulary is reusable even by teams that ignore the detector.
DRIFT itself is a three-stage audit. Claim Keeper builds a ledger of every claim the agent introduces, recording when it appears and where it gets reused. Support Seeker checks each consequential claim against trajectory evidence and labels it directly supported, weakly supported, missing, or conflicting. Dependency Tracer keeps only the risky claims that actually propagate into the final answer. The design bet is that auditing grounding beats judging plausibility.
The results, all with Claude-Sonnet-4.6 as auditor: 54.91% macro-F1 overall, 60.00% on the easy split, 47.28% on the hard split, up to 30 percentage points above handing the same model the raw trajectory. First-error accuracy peaks at 24.10%. The paper’s own framing is honest: sub-55% F1 is a research baseline, not a release gate.
Arbor: in-run prevention with hypothesis trees
Arbor attacks a different failure: an agent that keeps polishing the wrong thing. Its answer to “what should persist across a multi-day research run” is a hypothesis tree. Each node binds a hypothesis to an artifact version, experiment evidence, a score, and a distilled insight for later branches. A long-lived coordinator decides which frontier nodes to expand and which to prune; short-lived executors test one hypothesis each in isolated worktrees and return compact reports.
The mechanism that makes the tree more than a log is the merge gate. Development feedback is used for search, but a branch is promoted only when its improvement transfers to a held-out evaluator. A branch can look excellent during exploration and still be rejected, which is the paper’s defense against metric chasing.
The numbers come in two groups. On six Autonomous Optimization tasks spanning model training, harness engineering, and data synthesis, Arbor reports the best held-out result on all six, with more than 2.5x the average relative held-out gain of Codex and Claude Code under the same task interface and resource budget. On MLE-Bench Lite with GPT-5.5, Arbor reaches 86.36% Any Medal, the strongest result in its comparison. The ablation with Claude Opus 4.6 is the cleanest evidence that the structure does the work: full Arbor scores 81.82% Any Medal, dropping the tree leaves 63.64%, and dropping the insight feedback leaves 54.54%. Cost is logged rather than hidden: six completed runs consume between 20.12M and 43.19M tokens, which the authors describe as comparable to single-trajectory baselines.
Where the two loops meet
Three overlaps matter more than the packaging differences. First, both treat agent history as structured state rather than a chat transcript: semantic spans in one case, tree nodes in the other. Second, the two papers share a ruler, because Codex and Claude Code appear in DRIFT’s agentic baselines and in Arbor’s capability comparison. Third, both price their instrumentation: DRIFT reports an accuracy-versus-cost trade-off for its multi-stage audit, and Arbor logs token budgets per task. Reliability work that hides its cost is not yet reliability engineering.
The deeper agreement is philosophical: final answers are lossy summaries, and the inspectable object is the process. Where they diverge is position in the loop and burden of proof. DRIFT tolerates noisy judgment (what counts as harmful was decided by annotators with LLM assistance) and pays nothing during the run. Arbor demands executable evaluators and a dev/test split, and pays on every branch. DRIFT’s ceiling is a detector that is right about half the time; Arbor’s ceiling is a merge gate that only works where held-out evaluation means something.
Key numbers
| Dimension | DRIFT (TELBench) | Arbor (AO tasks + MLE-Bench Lite) |
|---|---|---|
| Role in the loop | Post-hoc diagnosis of finished trajectories | In-run prevention via merge gate |
| Unit of state | Semantic span, 11.95 per trajectory on average | Tree node: hypothesis + artifact version + evidence + insight |
| Benchmark scale | 1,000 verified instances (600 easy / 400 hard) from 2,790 raw runs | 6 AO tasks spanning training, harness engineering, data synthesis, plus MLE-Bench Lite |
| Headline number | 54.91% macro-F1 (Claude-Sonnet-4.6), up to +30 points over raw-trajectory reads | Best held-out result on all 6 AO tasks; 86.36% Any Medal with GPT-5.5 |
| Relative gain | Up to +30 percentage points macro-F1 vs the same model reading the raw trajectory | More than 2.5x the average relative held-out gain of Codex and Claude Code at the same interface and budget |
| Hardest sub-metric | First-error accuracy 24.10% | Ablation without insight feedback: 54.54% Any Medal |
| Ablation | All three stages contribute; each is load-bearing | 81.82% full vs 63.64% without tree vs 54.54% without insight feedback (Claude Opus 4.6) |
| Failure vocabulary | 18 primary faults in 6 families | Held-out gate against dev-set metric chasing |
| Cost accounting | Multi-stage audit with reported accuracy-vs-cost trade-off | 20.12M to 43.19M tokens across six cost logs |
| Shared reference points | Codex and Claude Code as agentic readers | Codex and Claude Code as capability baselines |
Same run, two readings: the table is not a single leaderboard, and that is the point. DRIFT’s percentages measure how well one model audits another model’s work; Arbor’s percentages measure whether a gated process out-researches strong open-ended agents.
When to use which
- Your agent is in production and users report bad answers: DRIFT’s span audit is the entry point. Replay failing trajectories through the claim-auditing loop, then bucket what you find with the 18-fault taxonomy; the vocabulary outlives the detector.
- You are building or hardening a research harness: Arbor’s merge gate is the more direct borrow. Require that any candidate improvement transfers to a held-out evaluator before it can replace the incumbent artifact, and keep the tree auditable.
- You must justify reliability work to a team that only trusts leaderboards: DRIFT gives you a benchmark (TELBench) and a ceiling (sub-55% F1); Arbor gives you a protocol and a cost log. Pick the evidence style that matches your risk.
- You can afford both: the two compose naturally in principle. DRIFT-style traces tell you which spans fail most often; those spans become the hypotheses Arbor-style executors must test, and held-out results tell you whether the fix generalized.
Limits and open questions
DRIFT’s ceiling is low and stated: 54.91% macro-F1 overall, 47.28% on the hard split, 24.10% first-error accuracy. TELBench is bounded to a handful of frameworks and models on three QA-style benchmarks, and its judgment of what counts as harmful is baked in by the annotation process. Whether the taxonomy transfers to coding agents or multi-agent systems is untested. Arbor needs executable evaluators, a dev/test split, and artifacts that can be versioned, which excludes a large share of real research; its baseline comparison may depend on the exact interface, evaluators, and time limits; and as a living technical report its numbers are a strong project report rather than a settled standard. Neither paper prices a wrong action at deployment time, and neither proves transfer across benchmark families. The open question both leave behind: once diagnosis and gating are cheap, does agent reliability stop being the bottleneck, or does the bottleneck move to evaluator design?
FAQ
Which one makes my research agent more reliable, DRIFT or Arbor?
They fix different halves. DRIFT tells you where a finished run broke, so it pays off when you already have trajectories and recurring failures. Arbor decides which improvements count as progress during the run, so it pays off when you are iterating on an artifact. Agents that ship to users need the first; research harnesses need the second.
Is DRIFT’s 54.91% macro-F1 good enough to gate releases?
No, and the paper’s plain reporting of the number says as much. Flagging any harmful span half the time and the first error a quarter of the time is a debugging aid. Use it to prioritize human review, not to block deploys automatically.
Does Arbor’s 86.36% Any Medal mean it beats Codex and Claude Code at research?
On MLE-Bench Lite with GPT-5.5, that is the reported number, and the six AO tasks show more than 2.5x the average relative held-out gain over both systems at the same interface and budget. The claim is narrower than “better at research”: Arbor wins at this gated refinement loop on executable tasks with a held-out split.
How much do the two systems cost to run?
DRIFT runs after the fact, and the paper reports an accuracy-versus-cost trade-off across its three stages, so the cheapest configuration buys less accuracy. Arbor logs 20.12M to 43.19M tokens across six completed runs, described as comparable to single-trajectory baselines. Neither is free; both make the price visible, which is the point.