Agent Harness vs Model: Why the Wrapper Can Beat the Model
Three 2026 papers quantify the agent harness: swapping only the adapter moves OpenClaw from 19.1% to 73.4% Pass@1 on the same model, and RHO tunes a harness from unlabeled trajectories.
One layer, three papers
Ask two engineers what an “agent harness” is and you will get three answers. The useful definition for 2026: the harness is everything around the model that the model did not write, namely its instructions, tools, skills, workspace contract, patch extraction, stopping rules, and runtime budget. The model is fixed; the harness is the system you actually ship and iterate on. Three recent papers attack that layer from three directions, and together they make a case that is hard to wave away: the harness is a first-class variable, comparable in size to the model itself.
Code as Agent Harness is the map: a UIUC-led survey arguing code is not what agents produce but the runtime they act through, organized into three layers. Claw-SWE-Bench is the measurement: a 350-task benchmark that isolates the harness from the model and quantifies how much scores move when only the harness changes. RHO is the lever: a Microsoft-and-CityU method that improves the harness using only the agent’s own past trajectories, no labels anywhere.
Read the numbers below as three independent claims, not a leaderboard. Each comes from its own setup; what transfers is the conclusion that harness engineering deserves its own budget line.
Key numbers
| Paper | What it isolates | Headline numbers | Evidence base |
|---|---|---|---|
| Code as Agent Harness | The concept: harness as executable substrate | ~20 code-reasoning + ~20 code-acting systems tabulated; 3 layers, 5 domains, 7 open problems | Survey, no new benchmark |
| Claw-SWE-Bench | Harness vs model on identical tasks | OpenClaw 19.1% to 73.4% Pass@1 on the same GLM 5.1, adapter-only change; harness choice 27.4 pp vs model choice 29.4 pp; Lite-80 costs 22.9% of full-350 with mean Pass@1 0.643 vs 0.639 | 350 GitHub issue tasks, 8 languages, 43 repos |
| RHO | Harness improvement without ground truth | SWE-Bench Pro 0.59 to 0.78 in one round, no external grading; beats Meta-Harness 0.62 at matched single-round budget (103 vs 41 agent calls); dropping self-consistency collapses to 0.56, below the vanilla baseline | SWE-Bench Pro, Terminal-Bench 2, GAIA-2, fixed Codex agent |
The cell that should stop you is the first one. Nineteen point one to seventy-three point four is not a model upgrade, a bigger context window, or a new prompting trick. It is the same backbone wearing a different adapter. If your evaluation pipeline reports a single number per model name, this is the number it is hiding.
What the harness actually is
The survey’s reframing is worth absorbing before the benchmark numbers, because it explains why the effect is so large. Its claim: stop treating LLM-written code as a generated artifact and start treating code as the operational substrate the agent reasons, acts, and coordinates through. Three properties make code uniquely suited to that role. Code is executable, so intent becomes an outcome you can verify by running it. Code is inspectable, so intermediate state is visible instead of locked in weights. Code is stateful, so a long task gets a place to keep its working memory across steps.
The survey splits the field into three layers, and the split is the practical checklist. Layer 1 is the harness interface: code for reasoning (program-of-thought style externalization, formal verification, iterative code-grounded reasoning), code for acting (skill selection, policy generation, Voyager-style lifelong skill accumulation), and code for environment (state representations, execution traces, evaluation environments). Layer 2 is the harness mechanisms that keep an agent alive over a long horizon: planning, six types of memory, four families of tool use, and Plan-Execute-Verify control loops. Layer 3 is scaling: multiple agents coordinating over a shared codebase, where the code itself becomes the synchronization substrate.
The one distinction in the survey that pays rent in practice is between agent-initiated code artifacts (what the model writes mid-task) and system-provided infrastructure (the harness the model runs inside). Most debugging sessions blur the two and waste hours optimizing the wrong one. The Claw-SWE-Bench and RHO numbers below are, in effect, repeated demonstrations of why the infrastructure half deserves its own attention.
The measurement: the harness moves scores almost as much as the model
Claw-SWE-Bench starts from a contract problem. SWE-bench expects a clean Docker workspace, a patch, and a prediction format. A general-purpose agent can be excellent at tool use and still fail the contract: no reliable patch extraction, no workspace management, wrong stopping point. The benchmark adds an adapter protocol so heterogeneous harnesses run under the same prompt, runtime budget, workspace rules, patch extraction, and evaluator; the harness becomes a controlled variable instead of invisible plumbing.
The results make the case bluntly. On the full 350-instance benchmark, OpenClaw on GLM 5.1 scores 19.1% Pass@1 with a minimal direct-diff adapter and 73.4% with the full adapter. Same model, same tasks, 54-point swing from adapter design alone. The sweeps put the harness axis next to the model axis: across nine models inside one harness, model choice moves Pass@1 by 29.4 percentage points; across five harnesses with models fixed, harness choice moves it by 27.4. Harness variation is nearly model-sized, and a leaderboard that hides the harness behind a model name ranks the wrong thing.
Two design details are worth stealing. First, Claw-SWE-Bench Lite: an 80-instance subset selected by a cost-aware, rank-aware procedure over 17 calibration columns, covering 34 of 43 repositories. It costs about 22.9% of a full run while preserving aggregate behavior, with mean Pass@1 of 0.643 on Lite versus 0.639 on the full set, a 0.4-point gap. That is the screening tool you want for harness iteration: cheap enough to run on every change, principled enough not to lie to you. Second, future-commit cleanup: after removing repository history that leaks post-issue information, no run improves, and the largest drop is Claude Opus 4.7 falling from 84.7% to 76.7%. Contamination treatment changes rankings, so prefer the cleaned numbers wherever a benchmark offers them.
The lever: improving the harness when you have no labels
If Claw-SWE-Bench tells you the harness matters, RHO tells you what to do about it on a Tuesday, when you have a pile of past agent runs and no graded validation set. Label-hungry harness optimizers propose an edit, score candidates on held-out labels, and keep the best. RHO replaces the labels with the agent’s own judgment over its rollouts, in three stages.
Coreset selection. Pick a small, difficulty-diverse subset of past tasks with a determinantal point process that balances two pulls: hard tasks, and spread across the task space. The paper’s analysis is worth noting because both one-sided versions fail: difficulty-only selection clusters on one task type, coverage-only wastes budget on easy tasks, and each trails even random sampling. Only the combined DPP reaches the top gain.
Diagnosis from rollouts. Re-solve each coreset task several times in parallel and extract two label-free signals. Self-validation reads a single trajectory for the agent’s own failed checks; self-consistency reads across the parallel trajectories for disagreement on the same task. Neither needs an answer key.
Best-of-N proposal and self-preference. Sample several candidate harnesses from the diagnoses, re-run them, and ship the one the agent prefers in pairwise comparison against the old harness. The ablations show every piece is load-bearing: dropping self-consistency collapses SWE-Bench Pro to 0.56, below the 0.59 vanilla baseline it was supposed to improve; dropping self-validation gives 0.70, and feeding raw trajectories with no diagnosis gives 0.60. And single proposals are too stochastic to trust, which is why sampling several and filtering matters.
The payoff is concrete in two ways. On scores: one round lifts SWE-Bench Pro from 0.59 to 0.78 with no external grading, versus 0.62, 0.61, and 0.64 for feedback-free memory-and-skills baselines at a matched agent-call budget, and versus 0.62 for Meta-Harness, a label-using optimizer, at matched single-round budget (RHO spends 103 agent calls to Meta-Harness’s 41; pushed to ten rounds and roughly 3x compute, Meta-Harness reaches 0.80 but still needs labels RHO never touches). Gains on Terminal-Bench 2 (0.71 to 0.76) and GAIA-2 (0.29 to 0.37) are real but smaller, so SWE-Bench Pro is the favorable case. On content: the optimized harness is not an abstract prompt tweak. The agent learns the Go toolchain sits outside its default path and that Python cache directories must be stripped before producing a diff or patches fail to apply, and it writes a check_build_and_lint tool encoding both. Those are debuggable engineering fixes, extracted automatically from failure data.
When to use which
The three papers compose into a sequence rather than competing with each other.
- Audit the contract first. Before spending on optimization, make sure your harness satisfies the task contract: patch extraction, workspace hygiene, stopping rules. Claw-SWE-Bench’s 19.1% number is what a capable model looks like inside a harness that fails the contract. No optimizer fixes a missing contract.
- Instrument the two axes separately. Report model scores inside a fixed harness and harness scores with models fixed. If you only track one number per model name, you cannot tell a model regression from a harness regression.
- Build a cheap regression loop. A Lite-style subset, selected for rank preservation rather than friendliness, run on every harness change. Paying 22.9% of full cost to catch a 50-point adapter regression is the best trade in this whole page.
- Then optimize, label-free if you must. If you have scored rollouts, label-using optimizers remain the most sample-efficient route. If you do not, RHO’s recipe works, but budget for its real costs: the diagnosis signals are compute (103 agent calls per round), self-preference can reward edits the model likes over edits that generalize, and the method assumes a base model competent enough that its self-consistency carries signal. On a weak model whose rollouts disagree for the wrong reasons, the diagnosis misleads, and the paper does not test that boundary.
Limits and open questions
All three papers have honest ceilings. Claw-SWE-Bench’s results are mostly single-run aggregates, and its sweeps (five harnesses, two models for the harness axis) prove the harness matters without mapping every model-harness interaction; provider pricing and cache accounting also move its cost numbers. RHO’s headline is one round on one benchmark with a fixed agent, its smaller gains elsewhere (+5 and +8 points) are the more typical case, and self-grading agents can approve harness edits that flatter them; the chosen candidate in RHO’s own Table 3 is not always the true top scorer. The survey is a map, not evidence: its executable-inspectable-stateful framing is clean but not falsifiable, and filing older tool-use work under a new label does not by itself make the label right. The open question all three leave hanging is whether the model-harness non-separability Claw-SWE-Bench shows for coding tasks holds for browser agents, desktop agents, and research agents; nobody has benchmarked those axes yet.
FAQ
What is an agent harness?
Everything around the model that the model did not write: instructions, tools, skills, workspace contract, patch extraction, stopping rules, and runtime budget. The survey’s framing is that code is the harness’s natural substrate because it is executable, inspectable, and stateful.
How much does the harness affect coding-agent scores?
A lot. On Claw-SWE-Bench, OpenClaw on the same GLM 5.1 backbone goes from 19.1% to 73.4% Pass@1 when only the adapter changes. In controlled sweeps, harness choice moves Pass@1 by 27.4 percentage points, close to the 29.4 points that model choice moves it.
Can I improve an agent harness without ground-truth labels?
Yes. RHO does it from unlabeled past trajectories: select a hard, diverse coreset with a DPP, diagnose failures with self-validation and self-consistency, propose candidate harnesses, and ship the one the agent prefers in pairwise comparison. One round lifted SWE-Bench Pro from 0.59 to 0.78 with no external grading.
What is the cheapest way to iterate on a harness?
A cost-aware Lite subset, in the spirit of Claw-SWE-Bench Lite-80: 80 instances selected over 17 calibration columns cost about 22.9% of the full 350-instance run while tracking its aggregate Pass@1 within 0.4 points. Run it on every harness change.
One line: the harness is not plumbing; it is a first-class variable worth 27 points of Pass@1, and it can be tuned from your own unlabeled trajectories. Read the papers: Code as Agent Harness, Claw-SWE-Bench, and RHO.