AI Agents · AI for Science · Vision-Language-Action
Scientific Agents Compared: Eywa vs LabVLA vs Crafter
Each system owns one segment of the scientific loop: Eywa calls domain foundation models (0.6154 to 0.6558 utility), LabVLA runs lab protocols (71.1% ID), Crafter drafts editable figures (50.34 vs 11.13).
The question: what should an agent do inside a lab?
“AI for science” gets discussed as one thing, usually as “the AI scientist.” These three 2026 papers are more precise. Each claims a single segment of the scientific loop, builds the narrow machinery that segment needs, and reports numbers only for that segment. Eywa handles reasoning over scientific data: time series, tabular records, and domain inputs that lose information when flattened into a prompt. LabVLA handles physical execution: turning written laboratory protocols into robot trajectories. Crafter handles one slice of communication: producing scientific figures that a human can still edit after review. None of the three decides which experiment to run next; that is the layer above them, where environment-engineering systems such as EurekAgent operate.
The shared bet is the interesting part. All three freeze a strong general-purpose model and spend their effort on the interface around it: a learned bond to specialist models, a simulation engine that manufactures training data, or a typed specification that five agents negotiate over. None of the headline numbers comes from swapping in a bigger foundation model.
Eywa: the reasoning layer
Eywa starts from an observation most agent frameworks skip: an LLM agent reads everything as text, so a 500-point time series or a wide medical table gets serialized into tokens before the model ever sees it. The paper formalizes this as a language interface bottleneck and argues, with an information-theoretic bound (Proposition 13) and a strict optimal-risk theorem (Theorem 3), that serialization alone caps how well a language-only agent can do on such inputs.
The machinery is a learned connection between the LLM and a domain foundation model, named Tsaheylu, with three parts: a query compiler that translates the agent’s state into a runnable configuration, a response adapter that converts numeric output back into something language-compatible, and a control policy that decides whether calling the specialist is worth it at all. The specialists are exposed as remote services through MCP, and in the paper the wired-in models are Chronos for time series and TabPFN for tabular data. Three tiers follow: EywaAgent (one LLM plus one foundation model), EywaMAS (a fixed multi-agent topology), and EywaOrchestra (a conductor that picks agent types, models, and topology per task).
All evaluation is on EywaBench, built from DeepPrinciple, MMLU-Pro, fev-bench, and TabArena, spanning 9 scientific subdomains and three modalities on a unified 0 to 1 utility scale. The single-agent variant lifts overall utility from 0.6154 to 0.6558 while using about 30% fewer tokens and roughly 10% lower latency. EywaMAS reaches 0.6761 against 0.6294 for homogeneous debate-style baselines, and EywaOrchestra nearly matches it at 0.6746 while cutting cost back to 8,335 tokens and 48.16s versus 11,214 tokens and 72.11s for the multi-agent variant. The per-domain split is where the honesty lives: social science reaches 0.7488 and physical science 0.6914, while life science sits at 0.5001, so the approach does not rescue every field. The backbones throughout are small models (gpt-4.1-nano, gpt-5-nano, gpt-5-mini), which keeps the benchmark cheap and leaves the frontier-model question open.
LabVLA: the hands
LabVLA’s bet is that laboratory robotics is a data problem before it is a modeling problem. Household VLA policies learn from mugs and tabletops; labs contain protocols, transparent liquids, and tasks where the right action in the wrong order invalidates the experiment. So the paper builds RoboGenesis, a simulation and data engine with 16 robot platforms, a 2,947-asset LabAssetLibrary, 10,000 generated lab scenes, and six-axis domain randomization, exporting only successful rollouts with structured annotations.
The model itself couples a Qwen3-VL-4B-Instruct backbone, made action-aware through FAST action token pretraining, to a flow-matching DiT action expert, with a stop-gradient path the authors call knowledge insulation to keep action loss from damaging the backbone’s language and vision knowledge. On LabUtopia, the model reports 71.1% average success in distribution and 70.0% out of distribution across six laboratory tasks, beating the next best policy, pi0, by 7.8 and 6.8 points, and the ID-to-OOD drop is only 1.1 points. The task split is less comfortable: Press Button reaches 100% ID, but Pour Liquid reaches only 43.3% ID and 34.2% OOD, and no baseline in the table exceeds 50% on that simulated column. The real-robot study transfers four tasks to a Franka platform with 50 rollouts per setting, landing roughly in the 76% to 92% range, but DreamZero or pi0.5 match or beat LabVLA on some rows, so the simulation lead does not automatically carry to the bench.
Crafter: the voice
Crafter treats a scientific figure as structured state rather than one prompt. Five agents loop over a shared, typed specification: an Intent Reasoner seeds the spec from text, a sketch, or a partial figure; a Plan Generator proposes multiple visual framings; a Critic returns per-dimension diagnostics (position, color, text, layout) as directives instead of a scalar score; a Specification Refiner writes typed edits back; and a Convergence Judge decides whether to accept, refine, or revert. Because the agents negotiate over a spec rather than pixels, the same harness covers three figure types under four input conditions, including mask completion and sketch refinement.
The numbers make the harness-over-model argument concrete. On PaperBanana-Bench, Crafter scores 50.34 against 33.73 for the PaperBanana agentic baseline, 11.13 for the same Nano Banana 2 backbone run alone, and 1.37 for GPT-Image-2. On CraftBench the margin is 50.20 against 28.00 for the strongest agentic baseline. A companion system, CraftEditor, converts the raster output into coordinate-faithful editable SVG at 8.04 out of 10 edit fidelity versus 6.91 for the prior best editor, with text (7.61) and arrows (7.83) as the weakest axes. Two caveats keep these numbers honest: everything is scored by a three-VLM ensemble (80 samples), and the benchmark tops out near 50 out of 100, which the papers’ own framing reads as “usable draft,” not submission-ready.
Where the three loops meet
Three agreements run across the papers. First, the harness thesis: each system’s gain comes from structure wrapped around a frozen model (the compiler-adapter-router bond, the data engine plus action expert, the five-agent spec loop), not from scale. Second, the ceiling discipline: each paper reports the slice where its system still fails, Eywa’s life-science 0.5001, LabVLA’s 34.2% OOD pouring, Crafter’s 7.61 text axis, instead of hiding the tail. Third, the pricing styles differ in a telling way: Eywa reports tokens and latency per configuration, LabVLA prices its claims in rollouts (50 per setting on the real Franka), and Crafter’s loop cost per figure is the number the paper does not foreground.
The disagreements matter too. Eywa’s utility scale rewards getting more of the task right anywhere on 0 to 1, LabVLA’s success rate is pass or fail per rollout, and Crafter’s figures are judged by other VLMs, which can reward looking right over being correct. These are three different contracts with the reader, and a team choosing among them is really choosing which contract matches its risk.
Key numbers
| Dimension | Eywa (reasoning) | LabVLA (execution) | Crafter (communication) |
|---|---|---|---|
| Segment claim | Reasoning over time series, tables, domain data | Protocol execution on lab robots | Editable figures, posters, infographics |
| Headline number | 0.6558 overall utility vs 0.6154 plain LLM agent | 71.1% ID / 70.0% OOD average success on LabUtopia | 50.34 on PaperBanana-Bench (0 to 100) |
| Same-harness gain | ~30% fewer tokens and ~10% lower latency at higher utility | +7.8 / +6.8 points over best baseline pi0 (63.3 / 63.2) | +39 points over the same Nano Banana 2 backbone alone (11.13) |
| Weakest reported slice | Life science 0.5001 utility (per-domain, gpt-5-nano) | Pour Liquid 43.3% ID / 34.2% OOD; no baseline above 50% | CraftEditor text axis 7.61/10, arrows 7.83/10 |
| Efficiency tier | EywaOrchestra 0.6746 utility at 8,335 tokens / 48.16s vs EywaMAS 0.6761 at 11,214 tokens / 72.11s | ID to OOD drop of 1.1 points | Five-agent loop per figure; cost not foregrounded |
| Judge and protocol | Unified 0 to 1 utility on EywaBench, 9 subdomains, 3 modalities | Simulation LabUtopia plus 4 real Franka tasks, 50 rollouts per setting | Three-VLM ensemble judging, 80 samples |
| Frozen component it wraps | Chronos and TabPFN via MCP | Qwen3-VL-4B backbone plus DiT action expert | Nano Banana 2 image model |
Three different rulers: the table lines up numbers from three papers that do not share a benchmark, a task, or a scoring contract, so no column is directly comparable with any other. Read each column against its own protocol, and read the bottom row as the shared design pattern rather than a metric.
When to use which
- Your bottleneck is reading scientific data (signals, tables, domain records): start with Eywa’s interface argument. If your LLM agent is serializing numbers into prompts, the paper’s compiler-adapter-router split and its 0.6154 to 0.6558 result show what a learned bond buys before you retrain anything.
- Your bottleneck is physical execution of known protocols: LabVLA is the reference. Its recipe (simulation engine for data, FAST pretraining, flow-matching action expert) is the part worth borrowing, and its 1.1-point ID-to-OOD drop shows what domain randomization buys. Budget for the fact that pouring-scale tasks remain open.
- Your bottleneck is producing figures a reviewer can edit: Crafter is the direct fit. The five-agent spec loop is framework-shaped rather than model-shaped, so it ports to other image backbones, and CraftEditor answers the question every generated-figure pipeline dodges, namely how the human fixes the output afterward.
- You need all three: they compose in principle, an Eywa-style agent can plan with domain models, a LabVLA-style executor can run the protocol, and Crafter-style output can report it. But composition is not demonstrated by any of the three papers, and each one quietly assumes a human above it specifying what science to do.
Limits and open questions
Eywa’s limits are structural: only two foundation models and two non-text modalities are wired in, orchestration picks from a finite topology pool with an LLM-mapped conductor rather than a learned policy, and the small backbones leave it unknown whether frontier models that serialize numbers better shrink the gap the interface is built to close. LabVLA’s real-robot evidence is four benchtop tasks on one robot, still far from wet-lab variability such as transparent liquids, contamination, and calibration drift, and the benchmark abstracts away scientific judgment entirely. Crafter’s scores cap near 50 out of 100 under VLM judging, its five-agent loop cost is not priced, and its SVG extraction is weakest exactly where scientific figures are hardest, text and arrows. The shared open question is the layer above all three: once reasoning, execution, and reporting each have a competent agent, who verifies that the experiment itself was worth running?
FAQ
Which one is the best AI scientist, Eywa, LabVLA, or Crafter?
None of them is an AI scientist, and that is the useful takeaway. Eywa scores 0.6558 utility on scientific reasoning over data, LabVLA executes 71.1% of lab protocols in distribution, and Crafter produces figures worth 50.34 out of 100 on its benchmark. They cover three different segments of research work, so the right question is which segment blocks your workflow, not which system wins a leaderboard.
Is Eywa’s 0.6558 vs 0.6154 utility improvement worth the added machinery?
On EywaBench, the single-agent variant reaches 0.6558 utility against 0.6154 for a plain LLM agent while using about 30% fewer tokens, so the paper shows both accuracy and efficiency moving the right way on the same benchmark. The catch is the per-domain split: life science sits at 0.5001 while social science reaches 0.7488, and the tests run on small backbones, so treat the average as a proof of the interface concept rather than a production guarantee.
Does LabVLA’s 71.1% success rate mean lab robots are solved?
No. The 71.1% is an in-distribution simulation average across six LabUtopia tasks, and the paper’s own hardest column, Pour Liquid, drops to 43.3% ID and 34.2% OOD with no baseline above 50%. The real Franka study lands in the 76% to 92% range across four tasks, but DreamZero and pi0.5 match or beat LabVLA on some rows. The result supports protocol execution under constrained settings, not wet-lab autonomy.
How much of Crafter’s 50.34 comes from the agents rather than the image model?
Almost all of it, by the paper’s own ablation of system designs: the same Nano Banana 2 backbone scores 11.13 alone and 50.34 inside the five-agent loop, a 39-point jump with no change to the image model. The benchmark ceiling near 50 out of 100, the VLM-judge protocol, and the 7.61 text-axis score on CraftEditor are the honest boundaries of that claim.
Which setup should I run if I want Eywa’s multi-agent gains without the token cost?
The paper’s own comparison answers this: EywaMAS reaches 0.6761 utility at 11,214 tokens and 72.11s, while EywaOrchestra gets a statistically similar 0.6746 at 8,335 tokens and 48.16s by having a conductor pick the topology per task. If utility per token matters more than the last 0.0015 of utility, the orchestration tier is the configuration the paper points you to.