AI Agents · Language Models · LLM Reasoning
Agents' Last Exam vs Pi-Bench: Two Gaps in Agent Capability
ALE tests whether agents finish explicit professional work (1,490 verifiable tasks, 2.6% pass on the hardest tier); Pi-Bench tests whether assistants catch what users never said (best proactivity 67.0%).
The question both benchmarks ask, and why one number can’t answer it
Every few months a vendor claims its agent can “do real work.” Two 2026 benchmarks set out to make that claim checkable, and they both conclude agents fall far short, but they locate the failure in two different places, and that difference is the whole point of comparing them.
Agents’ Last Exam (ALE) tests the explicit task: here is a professional job with a verifiable artifact, can the agent produce it? It is built from 1,490 task instances covering 960 expert-authored workflows across 55 digital industries. Pi-Bench tests the implicit intent: the user asked for one thing, but a good assistant would have handled several things the user never said out loud. Did the agent notice? It is built from 100 multi-turn tasks over five professional personas.
An agent can pass the first test and fail the second (it clears the checklist while ignoring everything implied), or fail the first while looking proactive (it asks clever clarifying questions but cannot produce the deliverable). If you are choosing a benchmark, or reading a vendor’s leaderboard claim, the first question is which of the two gaps the number actually measures.
Agents’ Last Exam: the explicit-task ceiling
ALE, from UC Berkeley, is built around a simple complaint: existing agent benchmarks are too narrow to say anything about economically valuable work. Terminal-Bench tests terminal tasks, SWE-bench tests software issues, WebArena and OSWorld test browser or desktop control. ALE instead maps professional digital work broadly: engineering design, scientific computing, finance, media, operations, and legal-style document work, grouped into 13 industry clusters and 55 subdomains.
Three design principles define it. Representativeness means tasks come from domain experts, not templated generation. Complexity means a task should feel like a scoped professional job: open tools, inspect files, run domain software, produce an artifact, not a one-line quiz. Verifiability is the hardest constraint: wherever a deterministic check exists (files, numbers, output artifacts, simulations, constraint satisfaction), ALE uses it, and tasks that can only be graded by asking another model whether the answer “seems good” are rejected or re-engineered. To control contamination, most of the pool is private: only about 150 of the 1,490 task instances are public, with the score-bearing rest kept for rolling evaluation.
The results are harsh. The hardest tier averages a 2.6% full pass rate. On ALE-CLI, the paper reports Codex with GPT-5.5 at 25.2% overall (41.5% on near-term tasks, 20.0% on full-spectrum tasks, and 4.5% on last-exam tasks), despite the same system reaching 82% on Terminal-Bench in the paper’s comparison. That 82%-to-25.2% spread across task types is the single most quoted number in the paper, and it is a statement about benchmark breadth, not about one model being bad. Runs are expensive: roughly $3 to $10 each, often tens of minutes to hours, capped at five hours.
The failure analysis matters more than the leaderboard. For Claude Code with Opus 4.7, roughly three quarters of failed cases are understanding and approach failures: the agent misreads the task, picks the wrong plan, or fails to recognize which domain-specific tool or workflow the job requires. ALE’s authors also observe that more compute and longer budgets do not reliably buy better performance, that among well-engineered systems model choice explains a much larger spread than harness choice, and that agents default to ad hoc scripts instead of the intended professional software, behavior a narrow coding benchmark rewards and a real workflow punishes.
Pi-Bench: the unstated-intent gap
Pi-Bench inverts the usual test. Every one of its 100 tasks carries an initial request, a set of hidden intents (latent requirements the user never spells out), and a checklist of verifiable deliverables. A purely reactive agent can pass the checklist while missing every hidden intent, which is exactly what the benchmark is designed to catch, because in a long assistant relationship the thing users notice is whether the assistant understood what they meant, not whether it executed what they typed.
The benchmark runs across five domain personas (Researcher, Marketer, Law Trainee, Pharmacist, Financier), each as an episode of 20 sessions inside a persistent project environment where artifacts accumulate. Six dependency groups of two to three tasks each force the agent to carry context forward. Two scores define performance. Proactivity is the fraction of hidden intents the agent resolves on its own, either by completing them directly or by asking a focused question that surfaces the requirement. Completeness is whether the verifiable checklist was satisfied, graded by rule-based or rubric graders. Splitting the two is the paper’s contribution: it separates “did the work” from “understood the person.”
The headline numbers: GPT-5.4 leads proactivity at 67.0% (±2.1) with 65.6% completeness; Claude Opus 4.6 posts 65.5% proactivity and the best completeness at 67.6%; Qwen3.6 Plus follows at 64.0% / 64.1%. Proactivity spans 43.1% to 67.0% across the nine models tested, a 24-point spread, while completeness sits in a tighter 52.1% to 67.6% band, and no frontier model clears 70% proactivity. The two skills visibly come apart in individual models: Kimi K2.5 scores 61.6% completeness but only 43.1% proactivity (it clears checklists reactively), while Seed2.0 Pro is the mirror image at 58.4% proactivity and 52.1% completeness.
Two finer-grained findings carry the practical weight. First, memory is doing the anticipating: stripping prior sessions out of the dependency groups cuts proactivity by 9.5 points on average but completeness by only 2.5; GPT-5.4 falls from 78.5% to 64.9% proactivity while its completeness barely moves. Today’s “proactive” agents are largely exploiting accumulated context, not modeling a persistent user. Second, domain gaps are large: legal handoffs show 84.1% completeness against 38.1% proactivity (the widest gap), while drug-design tasks flip it at 84.9% proactivity but 68.0% completeness; proactivity is domain-dependent, not a single scalar trait.
Key numbers
| Measurement | Agents’ Last Exam | Pi-Bench | Note |
|---|---|---|---|
| What is tested | Finishing explicit professional tasks | Resolving unstated user intents | Different gaps; not competing claims |
| Task scale | 960 workflows → 1,490 task instances, 55 industries, 13 clusters | 100 tasks, 5 personas, 20-session episodes | ALE is broader; Pi-Bench is deeper per persona |
| Difficulty structure | Near-term / full-spectrum / last-exam tiers | Hidden intents vs explicit checklist, 6 dependency groups | Both split one score into separable axes |
| Grading | Deterministic artifact checks; LLM-judge rejected where a check exists | Rule-based or rubric graders on checklists; intent resolution logged | Both avoid pure vibes-based grading |
| Frontier headline | Hardest tier 2.6% full pass; Codex GPT-5.5 25.2% ALE-CLI (41.5 / 20.0 / 4.5 by tier) | GPT-5.4 67.0% proactivity, 65.6% completeness; Opus 4.6 65.5 / 67.6 | Different models, harnesses, and scales, so they are not cross-comparable |
| Context ablation | Not reported; context is inside the task, not across sessions | Removing prior sessions: proactivity −9.5 pts, completeness −2.5 (GPT-5.4 78.5→64.9) | Pi-Bench quantifies memory dependence |
| Skill split evidence | ~75% of failures are understanding/approach failures (Claude Code + Opus 4.7) | Kimi K2.5 61.6 Comp / 43.1 Proc vs Seed2.0 Pro 58.4 Proc / 52.1 Comp | ALE: agents mis-plan; Pi-Bench: skills dissociate |
| Domain variation | 55 industries with uneven depth per domain | Legal 84.1 Comp / 38.1 Proc; drug design 84.9 Proc / 68.0 Comp | Both warn against one leaderboard number |
| External benchmark anchor | Same Codex GPT-5.5: 82% Terminal-Bench vs 25.2% ALE-CLI | None reported | ALE’s breadth claim in one pair of numbers |
| Cost per evaluation | Roughly $3–$10 per run, tens of minutes to hours, 5-hour cap | Not reported | ALE is expensive to run at scale |
| Public availability | ~150 of 1,490 instances public; rest private against contamination | Not stated on the page | Check each paper’s release before replicating |
Read the table for what it cannot tell you: no model appears in both benchmarks’ headline results, no harness is shared, and the two scales (1,490 one-shot professional tasks vs 100 twenty-session episodes) answer different questions. Any ranking that merges them into one “agent capability” score is manufacturing a comparison the data does not contain.
What the two benchmarks agree on
Strip away the surface differences and the two papers make the same three claims. First, headroom is large and structural: 2.6% on ALE’s hardest tier and 67.0% proactivity at the frontier are both “nowhere near solved” numbers, arrived at from opposite directions. Second, context is load-bearing: ALE’s failure analysis shows agents collapsing when domain context must be recognized and used, and Pi-Bench’s 9.5-point memory ablation shows “proactivity” degrades sharply when accumulated context is removed. Third, model choice beats harness polish: ALE reports model choice explains more spread than harness choice among well-engineered systems, and Pi-Bench’s 24-point proactivity spread across nine models on a single scaffold says the scaffold is not the differentiator.
Where they diverge
The divergences are the useful part. ALE grades artifacts (did the file, the number, the simulation output check out); Pi-Bench grades behavior toward a person (did the agent do or ask about the thing that was never typed). ALE’s tasks are self-contained professional jobs; Pi-Bench’s are relational episodes where the whole point is that artifacts accumulate and prior sessions matter. ALE keeps its grading pool mostly private and contamination-controlled; Pi-Bench’s page does not describe a comparable private-pool design, and its users are simulated personas, so its hidden intents are what the authors decided a Researcher or Pharmacist would want, a reasonable proxy that can reward pattern-matching to the benchmark. Pi-Bench also runs on a single agentic scaffold adapted from Nanobot, so its model ranking is not guaranteed scaffold-independent. And the scopes are mirror images: ALE deliberately skips field work, embodied robotics, and long organizational projects, while Pi-Bench’s five-persona depth trades away open-world coverage.
When to use which
- You are building or buying a computer-use agent for professional work (coding, analysis, document production) and want an honest capability number: ALE-style verifiable professional tasks are the right instrument. Its 82%-to-25.2% Terminal-Bench contrast is a standing reminder that narrow benchmark scores overstate breadth.
- You are building an assistant product where users expect anticipation (scheduling, research support, ongoing projects): Pi-Bench-style hidden-intent probes are the right instrument, and its proactivity/completeness split is the diagnostic: an assistant that scores high on completeness but low on proactivity is a reactive tool wearing a proactive label.
- You can only run one cheap experiment: replay Pi-Bench’s memory ablation on your own logs: strip prior-session context from a sample of real sessions and measure how much “proactive” behavior disappears. The 9.5-point average drop in the paper is strong prior evidence that much of what looks like anticipation is context exploitation.
- You are reading a vendor leaderboard: check whether the number comes from explicit-task benchmarks (ALE family), intent benchmarks (Pi-Bench family), or narrow task suites (Terminal-Bench, SWE-bench). The ALE paper’s own contrast shows a system can look excellent on the third and still fail the first.
Limits and open questions
The central gap is that the two benchmarks never meet: different models, different scaffolds, different task scales, and different metrics, so no same-harness comparison exists between them; the table above is a comparison of protocols, not of agents. Both evaluate against simulated or expert-authored proxies rather than live users, so both can be gamed by systems tuned to the proxy. Pi-Bench’s single-scaffold design means its model ranking may not transfer to a different harness, tool set, or memory implementation; ALE’s private pool means outsiders cannot fully audit the score-bearing tasks. Neither benchmark measures long open-ended projects where success depends on people and institutions, and neither captures the cost dimension at deployment scale; ALE is expensive per run ($3–$10, up to five hours), Pi-Bench’s page reports no cost analysis. Finally, both are snapshots: model generations move fast enough that ALE’s Codex GPT-5.5 and Pi-Bench’s GPT-5.4 numbers will age, while the protocols (verifiable professional tasks, hidden-intent proactivity scoring, memory ablations) are the durable contributions.
FAQ
How do Agents’ Last Exam and Pi-Bench compare on the same tasks?
They do not share any tasks, models, or harnesses. ALE reports Codex with GPT-5.5 at 25.2% on ALE-CLI with a 2.6% hardest-tier average; Pi-Bench reports GPT-5.4 at 67.0% proactivity and 65.6% completeness across 100 tasks. The two sets of numbers measure different capabilities on different scales and cannot be ranked against each other.
What is the single most important result in each benchmark?
For ALE, the same Codex GPT-5.5 scoring 82% on Terminal-Bench but 25.2% on ALE-CLI (dropping to 4.5% on last-exam tasks) is the strongest evidence that narrow benchmark success does not transfer to broad professional work. For Pi-Bench, removing prior sessions from dependency groups cuts proactivity by 9.5 points while completeness drops only 2.5, which quantifies how much current “proactivity” depends on accumulated context.
Which benchmark should I use to evaluate my agent?
If your agent executes explicit professional tasks, build ALE-style verifiable tasks with deterministic artifact checks. If your product is an ongoing assistant, build Pi-Bench-style tasks with hidden intents and score proactivity separately from completeness. If you do both, keep the two scores apart; an agent can be strong on artifacts and weak on anticipation, and averaging the two hides the failure mode users will actually notice.
Are ALE and Pi-Bench public?
ALE is partially public: about 150 of its 1,490 task instances are released, with the rest kept private for rolling evaluation and contamination control. Pi-Bench’s public availability is not described on the benchmark’s page, so check the paper’s artifacts before planning a replication.
Do these benchmarks prove agents are useless?
No. ALE’s own failure analysis is more specific than that: roughly three quarters of failures are understanding and approach failures, meaning the agents misread tasks or miss which domain workflow applies, a different problem from raw tool capability. Pi-Bench shows frontier models resolving two thirds of unstated intents. Both benchmarks say agents are uneven, not useless, and both give engineers a way to find where the unevenness sits.