Proactive AI Assistants, Evaluated: Pi-Bench vs SoCRATES vs TIDE

The question all three papers are circling

“Proactive” has become the favorite adjective of 2026 assistant marketing. The word hides three very different capabilities: anticipating what a user wants but never typed, judging when to intervene in a human conflict, and surfacing problems nobody asked about. Three 2026 papers build a checkable test for each one, and a fourth, the Digital Colleague survey, supplies the systems-level frame that explains why all three are hard. Comparing them tells you which failure mode your own product actually has, because the headline numbers do not move together.

Pi-Bench: anticipation across a sustained relationship

Pi-Bench, from Shanghai AI Lab, tests whether an assistant resolves needs the user never stated. Each of its 100 multi-turn tasks carries an initial request, a set of hidden intents, and a verifiable checklist. Five professional personas (Researcher, Marketer, Law Trainee, Pharmacist, Financier) each run an episode of 20 sessions in a persistent environment, with six dependency groups that force context to carry forward. Two scores split the skill: proactivity (fraction of hidden intents resolved on their own) and completeness (checklist satisfied).

The results: GPT-5.4 leads proactivity at 67.0% (±2.1) with 65.6% completeness; Claude Opus 4.6 posts 65.5% proactivity and the best completeness, 67.6%. No model among the nine tested clears 70% proactivity, and the spread is 24 points (43.1% to 67.0%). The sharpest finding is the memory ablation: stripping prior sessions out of dependency groups cuts proactivity 9.5 points on average but completeness only 2.5, with GPT-5.4 falling from 78.5% to 64.9%. Most of what looks like anticipation is accumulated context being exploited. Domain gaps are equally large: legal handoffs show 84.1% completeness against 38.1% proactivity, while drug-design tasks flip the ratio (84.9% proactivity, 68.0% completeness). One caveat: everything runs on a single Nanobot-adapted scaffold with simulated users, so the ranking is not guaranteed to transfer.

SoCRATES: knowing when to step into a conflict

SoCRATES moves proactivity from task assistance to social judgment: should an LLM mediator intervene in a multi-party conflict, and how well does it adapt? Scenarios are generated from real conflicts through an agentic pipeline across eight domains, and models are probed on five socio-cognitive axes: strategic posture, party composition, history length, emotional reactivity, and cultural identity. Eight frontier LLMs are benchmarked, graded by a topic-localized evaluator that scores each topic only on the turns that advance it, which the authors build specifically to cut off-topic noise.

Two numbers carry the paper. The evaluator reaches 0.82 alignment with human experts, more than doubling a per-turn baseline, which matters because mediation quality is exactly the kind of thing naive per-turn grading gets wrong. And the strongest mediator closes only about a third of the unmediated consensus gap, with performance varying sharply across the socio-cognitive axes. In plain terms: even the best model is a noticeably incomplete substitute for human mediation, and its weakness moves depending on who the parties are and how emotionally loaded the history is.

TIDE: finding the problems nobody filed

TIDE tests a third sense of proactive: given a messy body of context, discover multiple hidden problems, each grounded in evidence and paired with a concrete action. Single-pass prediction anchors on the most salient issue and returns generic claims, so TIDE iterates in small batches conditioned on what has already been found, steering each round with thought templates distilled from previously solved cases (20 templates for the workspace setting, 108 for the repository setting; 10 rounds and 3 rounds respectively).

The testbeds are concrete: 30 personal workspaces holding 150 problems (4 to 6 problems and 88 to 113 candidate artifacts each), and 20 multi-bug software-repository instances from 11 projects holding 146 problems (2 to 41 problems, 6 to 646 candidate functions per instance). Four backbones are tested: GPT-5 mini, Claude Sonnet 4.5, Gemini 3.5 Flash, and Qwen 3.6 Flash. On the workspace setting, GPT-5 mini with TIDE reaches 69.06 coverage / 70.46 F1 on retrieval against 47.60 / 54.32 for the best single-agent baseline, an average +20.01 points across metrics (p < 0.001); Gemini 3.5 Flash does even better at 83.84 / 84.91, while Claude Sonnet 4.5 reaches only 32.01 / 37.01 on the same framework, showing the method amplifies an already-strong backbone rather than rescuing a weak one. The repository setting is much harder for everyone (GPT-5 mini: 16.82 coverage). Ablations back the design: removing the templates drops GPT-5 mini workspace retrieval to 62.48, and substituting few-shot demos manages only 10.40 repo retrieval coverage versus 16.82. The LLM judge itself is calibrated (89.2% exact agreement with humans versus 88.1% human-to-human, Fleiss κ = 0.80), and TIDE costs $0.444 per workspace run versus $0.442 for the multi-agent baseline it beats and $0.047 for the single agent it beats by 20 points.

The survey frame: why all three are hard

The Digital Colleague survey from Tencent YouTu Lab, Tsinghua, and others supplies the vocabulary the benchmarks implicitly assume. Its claim is that the qualitative leap is workspace plus skill: an agent that keeps persistent state (files, logs, permissions) and loads reusable procedures (a SKILL.md-style directory asset, more than a prompt, more than a tool) instead of improvising each time. Proactivity of the Pi-Bench and TIDE kind is only possible on top of that persistence; an agent with no memory of prior sessions cannot anticipate, which is exactly what the 9.5-point memory ablation quantifies.

The survey also stages evaluation into four levels, and the three benchmarks sit at different rungs: Stage I scores final answers, Stage II scores reasoning traces, Stage III scores task closure (did the environment end in the intended state, as SWE-bench and OSWorld do), and Stage IV adds workspace capability and safety. Pi-Bench is closest to a Stage IV relational test, TIDE is a Stage III-style state inspection, and SoCRATES grades interaction quality that none of the four stages cleanly covers. Treat the survey as a map, though: it ships no experiments, and its own limitations section admits that skills go stale on UI redesigns and dependency bumps, which is the same brittleness Pi-Bench’s memory ablation exposes from the benchmark side.

Key numbers

Measurement	Pi-Bench	SoCRATES	TIDE	Note
What “proactive” means	Resolving hidden user intents	Deciding when and how to mediate conflicts	Discovering unreported problems	Three different skills; one word
Scale	100 tasks, 5 personas, 20-session episodes	Scenarios from real conflicts, 8 domains	150 workspace problems (30 workspaces) + 146 repo problems (20 instances, 11 projects)	All small on purpose, graded by hand or rule
Backbone count	9 frontier models, one scaffold	8 frontier LLMs, names not reported	4 named backbones (GPT-5 mini, Claude Sonnet 4.5, Gemini 3.5 Flash, Qwen 3.6 Flash)	Only TIDE names its models
Headline result	GPT-5.4 67.0% proactivity, 65.6% completeness; no model over 70%	Evaluator 0.82 alignment with experts; best mediator closes ~1/3 of consensus gap	GPT-5 mini 69.06/70.46 retrieval Cov/F1, +20.01 pts vs single agent (p < 0.001); Gemini 83.84/84.91	Different metrics, not cross-comparable
Backbone spread	43.1% to 67.0% proactivity (24 pts) across 9 models	Varies sharply across 5 socio-cognitive axes	Workspace retrieval: Gemini 83.84 vs Claude 32.01 under identical TIDE	Spread means proactivity is not commoditized
Memory / context role	Removing prior sessions: proactivity −9.5, completeness −2.5	History length is one of the 5 stress axes	Iteration conditions on problems already found; templates from solved cases	All three show context is load-bearing
Grading design	Rule-based and rubric graders on checklists	Topic-localized evaluator, 0.82 expert alignment	LLM judge with 89.2% human agreement, κ = 0.80	All three invest in grader reliability
Cost signal	Not reported	Not reported	$0.444/run vs $0.442 multi-agent baseline vs $0.047 single agent	TIDE is priced like the baseline it beats
Public artifacts	Not described on the page	Not stated on the page	Not stated on the page	Check each paper before replicating

What the three benchmarks agree on

First, the frontier is far from the ceiling in every sense of the word: 67.0% proactivity at best, a third of the mediation gap at best, and repository-setting retrieval under 23% coverage for every backbone. Second, context is the substrate of proactivity: Pi-Bench quantifies it with the memory ablation, TIDE conditions each round on what has been found, and SoCRATES treats history length as a stress axis. Strip the context and the capability degrades disproportionately. Third, grader design is the research: each paper’s most durable contribution is how it scores (hidden-intent bookkeeping, topic-localized turn grading, template-guided iteration with a calibrated judge), not any single leaderboard number.

Where they diverge

The unit of proactivity differs. Pi-Bench grades behavior toward one user over time, SoCRATES grades judgment in a multi-party social setting, and TIDE grades coverage of a state nobody fully described. That means the benchmarks are complements, not competitors: a personal assistant product should run Pi-Bench-style probes, a mediation or negotiation feature needs SoCRATES-style axes, and an agent that monitors workspaces or codebases needs TIDE-style discovery. The failure modes also differ: Pi-Bench failures are memory and inference failures, SoCRATES failures are socio-cognitive, TIDE failures are coverage and anchoring failures. And the evidence types differ: Pi-Bench reports a controlled ablation, SoCRATES reports evaluator alignment against experts, TIDE reports significance-tested gains over four baselines, so decide which evidentiary style matches your risk before committing to one.

When to use which

• You are building an ongoing assistant (scheduling, research support, client work): use Pi-Bench’s proactivity/completeness split, and replay its memory ablation on your own session logs before trusting any “proactive” claim.
• Your product touches disputes, negotiations, or group decisions: SoCRATES’ five socio-cognitive axes are the checklist, and its 0.82 evaluator design is worth copying more than its leaderboard.
• Your agent watches a workspace, codebase, or ops surface for issues: TIDE’s iterative template loop is the method, and its workspace-versus-repository contrast is a warning: problem discovery in dense technical state is far harder than in personal context.
• You are designing the harness itself: the Digital Colleague survey’s workspace-plus-skill thesis and four evaluation stages give you the vocabulary for where persistence and skill libraries must live before any of the three benchmarks can even run.

Limits and open questions

All three benchmarks evaluate against simulated or authored proxies rather than live users, so tuned systems can overfit the proxy. Pi-Bench runs on a single scaffold, so its model ranking may not transfer. SoCRATES does not name its models, which limits reproducibility of the leaderboard. TIDE’s gains are largest on the setting where its backbones are already strong, and its repository numbers are low in absolute terms for everyone, so the framework’s ceiling in dense technical state is unclear. None of the three measures cost at deployment scale, latency, or the failure cost of a wrong proactive action, which is often worse than a missed one. And the survey’s workspace-plus-skill thesis remains unproven: no controlled experiment yet shows persistent state plus skill libraries beating a strong stateless baseline on task closure rate.

FAQ

Which benchmark is best for measuring proactive AI assistants?

None covers the whole concept. Pi-Bench measures hidden-intent anticipation in long assistant relationships (100 tasks, best proactivity 67.0% for GPT-5.4). SoCRATES measures mediation judgment in conflicts (best mediator closes about a third of the consensus gap). TIDE measures discovery of unreported problems (up to +20.01 points over single-agent baselines, p < 0.001). Match the benchmark to the capability you ship.

What is the single most important number across the three papers?

Pi-Bench’s memory ablation: removing prior sessions drops proactivity 9.5 points while completeness falls only 2.5, with GPT-5.4 going from 78.5% to 64.9%. It is the cleanest evidence that current “proactivity” is largely context exploitation, and it is cheap to replicate on your own logs.

Do any models do well on all three?

The papers do not share models or harnesses, so no cross-benchmark ranking exists. Within TIDE, Gemini 3.5 Flash leads the workspace setting (83.84 retrieval coverage) while Claude Sonnet 4.5 trails badly (32.01) under the identical framework, and Pi-Bench shows a 24-point proactivity spread across nine models. Proactivity is not a commoditized, model-agnostic trait.

Are these benchmarks public?

Pi-Bench’s public availability is not described on the paper’s page. SoCRATES reports its evaluator design (0.82 expert alignment) but its model list and artifacts are not stated. TIDE names its four backbones and baselines but the page does not detail release artifacts. Check each paper’s project page before planning a replication.

Does “proactive” mean the same thing in all three papers?

No, and that is the point. Pi-Bench: doing the unstated task for one user. SoCRATES: intervening well in a multi-party human conflict. TIDE: surfacing problems nobody filed. The Digital Colleague survey unifies them only at the infrastructure level: all three presume persistent workspace state plus reusable skills. Capabilities and infrastructure are different questions; benchmark the first, engineer the second.