Research Papers
EN ZH

Vision-Language-Action · World Models · Robotics

World Pilot: Steering a VLA Policy with World-Action Priors

World Pilot adds two world-model priors to a VLA policy and hits 84.7% total success on LIBERO-Plus zero-shot OOD, up 4.2 over the ABot-M0 base, and the scene prior works even from a world model never action-trained.

CASIA, Nanjing University, Beihang University · Paper published 2026-06-10 · 6 min read

World Pilot: Steering a VLA Policy with World-Action Priors

Quick answer

World Pilot reaches 84.7% total success on the LIBERO-Plus zero-shot out-of-distribution benchmark, 4.2 points above the ABot-M0 policy it wraps (80.5%). The gain does not come from a bigger or better-trained policy network. It comes from feeding the same policy two priors drawn from a separate World-Action Model (WAM): Latent Steering conditions the perception layer on a predicted scene-evolution latent, and Action Steering hands the action head an anticipated trajectory as a motion prior. Run alone, Latent Steering scores 83.7% and Action Steering 83.1%; together they reach 84.7%, so the two pathways add roughly independently. The surprising part is that the scene prior keeps working when it comes from a video-pretrained world model that was never post-trained on robot actions (82.6% on LIBERO-Plus, still +2.1 over the base).

Why static-pretrained VLAs miss contact dynamics

A vision-language-action model inherits its grounding from image-text pretraining, which is static. It knows what a towel looks like; it has no sense of how a towel deforms when you pull a corner. That gap shows up exactly where manipulation gets hard: contact-rich, multi-step tasks under novel viewpoints, object geometry, or deformable state. World Pilot’s premise is that a world model already carries that missing dynamics signal, because predicting the next frames forces it to model how the scene will move. So instead of retraining the policy to learn dynamics, the work injects a world model’s prediction into the policy at inference. This is the same instinct behind Kairos, which builds a native world-model stack for physical AI, but World Pilot keeps the policy fixed and treats the world model as a bolt-on advisor.

The two steering pathways

Latent Steering takes the WAM’s predicted future, a 5-step scene-evolution latent, and conditions the policy’s perception layer on it before the policy decides anything. Action Steering takes the WAM’s anticipated trajectory and feeds it to the flow-matching action head as a starting motion prior rather than letting the head generate from noise. The base policy here is ABot-M0 with a Qwen3-VL backbone and a DiT-based flow-matching action head; the WAM is Cosmos Policy running 5-step denoising.

The form of each prior matters more than you might expect. Feeding the future as a latent (84.7%) beats decoding it to an actual future image first and conditioning on that (83.5%), so the policy uses the compressed prediction better than a rendered one. On the action side, compressing the anticipated trajectory into a single encoded token (84.7%) beats per-step tokens (83.6%), flow initialization (84.1%), and the raw trajectory (83.0%). The lesson for builders: the priors help most as compact conditioning signals, not as literal images or step-by-step plans the policy has to parse.

How to read the 84.7%

84.7% is the total success rate on LIBERO-Plus specifically, a zero-shot OOD benchmark, not a clean-condition score and not a universal number. The honest comparison is same-policy-with-priors versus same-policy-alone, and on that axis +4.2 is the real headline. Per-axis the gains are uneven: World Pilot leads on appearance shifts (Light 98.6%, Background 96.4%, Noise 93.6%) and Camera viewpoint (82.8%, +13.2 over the next baseline), but Robot-embodiment shift stays at 60.6%, the weakest axis. So “SOTA total” is true while hiding that one category is still hard. Treat the headline as evidence that world-model priors transfer dynamics knowledge, not as a claim the policy now handles every shift.

The genuinely load-bearing result is the WAM ablation. A world model trained only to predict scenes, with no action post-training at all (Cosmos-Predict), still lifts the policy to 82.6% on LIBERO-Plus, 62.7% on RoboCasa (+8.7), and 85.3% on RoboTwin2.0 (+4.1). Adding action post-training (Cosmos Policy) buys only about another 1.1 points on LIBERO-Plus. That says most of the transferable signal is in plain video prediction, which is cheap and abundant, and the action-specific tuning is a small top-up. If that holds, you can steer a manipulation policy with an off-the-shelf video world model.

Key results

  • LIBERO-Plus total: 84.7% vs 80.5% for the ABot-M0 base (+4.2), reported as SOTA total success on the zero-shot OOD benchmark.
  • Pathway ablation: Latent Steering alone 83.7% (+3.2), Action Steering alone 83.1% (+2.6), combined 84.7% (+4.2), so the two add nearly independently and Latent is the slightly stronger single pathway.
  • No action post-training needed: a scene-only world model (Cosmos-Predict) still reaches 82.6% on LIBERO-Plus (+2.1), 62.7% on RoboCasa (+8.7), and 85.3% on RoboTwin2.0 (+4.1); adding action post-training adds only about 1.1 points.
  • Real-robot, 4 tasks: highest success on every in-distribution and shifted setting, e.g. Fruit-to-Plate 90% ID / 75% novel fruit / 70% layout shift, Fold Towel 85% / 75% / 70%; the ID-to-OOD drop stays within about 20 points while baselines drop 25-50.
  • Prior form: future-as-latent beats decoded future image (84.7 vs 83.5); single action token beats raw trajectory (84.7 vs 83.0), so compact conditioning wins over literal signals.
  • Per-axis: strongest on appearance (Light 98.6%, Background 96.4%) and camera viewpoint (+13.2), weakest on robot-embodiment shift (60.6%).

Limits and open questions

World Pilot inherits the WAM’s coverage. When a test scene falls outside the world model’s video-pretraining distribution, both priors degrade and the gains shrink, so the method is only as general as the world model behind it. The gains are uneven across LIBERO-Plus axes, and the robot-embodiment axis (60.6%) shows the priors do not fix everything. On real hardware the OOD success still drops 10-20 points, better than baselines but not solved. The biggest practical cost is runtime: a WAM forward pass per control step adds latency the paper says limits applicability to high-frequency reactive control, and there is no wall-clock or throughput figure to size that. The +1.1 from action post-training is small enough that whether it is worth the extra training pipeline is left open, and the work does not test how prediction errors in the WAM propagate into wrong actions.

FAQ

Does World Pilot need a world model that was post-trained on robot actions?

No, and that is the headline finding. A world model trained only to predict scene evolution from video, with no action post-training (Cosmos-Predict), still lifts the policy to 82.6% on LIBERO-Plus (+2.1 over the ABot-M0 base), 62.7% on RoboCasa (+8.7), and 85.3% on RoboTwin2.0 (+4.1). Adding action post-training (Cosmos Policy) buys only about another 1.1 points, so most of the transferable signal is in plain video prediction.

In World Pilot, does Latent Steering or Action Steering help more?

Latent Steering is slightly stronger as a single pathway: 83.7% (+3.2 over base) versus 83.1% (+2.6) for Action Steering. They combine to 84.7% (+4.2), which is close to the sum of their individual gains, so the perception-side scene prior and the action-side motion prior fix largely different failures rather than overlapping.

How much does World Pilot improve over the ABot-M0 baseline on LIBERO-Plus?

World Pilot reaches 84.7% total success versus 80.5% for the ABot-M0 policy it wraps, a 4.2-point gain on the zero-shot OOD benchmark. The same policy weights are used in both runs, so the gain is attributable to the world-action priors, not to a larger or retrained policy network.

What are World Pilot’s real-robot results across the four tasks?

Across Stack Blocks, Fold Towel, Fruit-to-Plate, and Container-Lid Alignment, World Pilot has the highest success on every in-distribution and shifted setting. For example Fruit-to-Plate scores 90% in-distribution, 75% on a novel fruit, and 70% under layout shift. The in-distribution to OOD drop stays within about 20 points, while comparison baselines drop 25 to 50 points.

One line: World Pilot shows a manipulation policy can borrow contact dynamics from a video world model at inference, turning an 80.5% LIBERO-Plus base into 84.7% without retraining the policy and without an action-trained world model. Read the original paper on arXiv.