Research Papers
EN ZH

Vision Foundation Models · World Models · Video Generation

VLM vs Video Generation: Which Pretraining Wins at Spatial Intelligence?

A frozen-feature probe pits VLMs against video-generation models on spatial tasks. VLMs win semantics (92.08 mAP vs 69.89), video models win geometry (0.527 camera AUC vs 0.330), and a naive concat of the two beats both.

Zhejiang University, Om AI Lab · Paper published 2026-05-27 · 6 min read

VLM vs Video Generation: Which Pretraining Wins at Spatial Intelligence?

Quick answer

Neither paradigm wins outright. Zhejiang University and Om AI Lab froze a batch of vision-language models (VLMs) and video-generation models (VGMs), then trained only a thin probe on top to read out three spatial skills: semantic tagging, cross-view instance grouping, and dense 3D geometry. The split is clean. VLMs dominate semantics, scoring 92.08 average mAP on ScanNet20 tagging against 69.89 for video models, and they hold instance grouping too (22.66 vs 13.24 T-mIoU). Video models flip the result on geometry, hitting 0.527 average camera AUC@30 on DL3DV against the VLMs’ 0.330, with lower depth error (0.072 vs 0.113 AbsRel). The headline is the third finding: concatenate one VGM and one VLM, train the same probe, and you get 92.30 mAP and 0.615 camera AUC, beating either backbone on both axes. So the practical answer is to stop choosing.

What the spatial benchmarks measure

The three tasks are picked to separate semantics from geometry, which is the whole point of the comparison. Semantic tagging asks which object categories appear across a clip, scored as mAP on ScanNet20. Instance grouping is harder: given multiple views, decide which pixels belong to the same physical object, scored as a temporal mIoU (T-mIoU) that rewards consistency across frames. The third axis, 3D geometry, bundles dense depth (AbsRel error), point-map reconstruction, and camera-motion recovery (AUC@30 on pose), measured on DL3DV scenes. Tagging and grouping are about what and which; geometry is about where and how the camera moved. Putting all three under one probe is what makes the VLM-vs-VGM gap legible instead of anecdotal.

If you care about the spatial-reasoning side specifically, our breakdown of cross-view spatial reasoning in VLMs covers why view changes are the hard case for language-aligned features.

VLM vs video-generation backbones, head to head

The lineup is deliberately broad on both sides. On the VLM bench: InternVL3 (1B/2B/8B), InternVL3.5 (4B/8B), Qwen2.5-VL (3B/7B), and Qwen3-VL (2B/4B/8B). On the video side: WAN2.1 (T2V 1.3B/14B and I2V-14B), CogVideoX (T2V 2B/5B and I2V-5B), OpenSora2.0, and Aether. Every model is frozen. Only the probe (an alternating frame-attention plus global-attention transformer) and the task heads get trained, so the numbers read out what each pretraining already encoded, not what a fine-tune could install.

Why does the split fall this way? Contrastive image-text pretraining teaches a VLM to name things, so its features carry object identity that transfers straight to tagging and grouping. A video generator is trained to predict the next frame, which forces it to model motion, occlusion, and parallax. That objective happens to encode camera geometry and depth far more cheaply than a captioning objective does. Qwen3-VL-2B posts the best semantic scores on its side; WAN2.1-T2V-14B leads geometry on the other. The two objectives are pulling representations in different directions, and the probe just exposes it.

Key results

  • Semantic tagging (ScanNet20, mAP): VLM average 92.08 vs VGM 69.89, a 22-point gap. VLMs also lead APmid (87.28 vs 58.63).
  • Instance grouping (T-mIoU / T-SR): VLM 22.66 / 11.23 vs VGM 13.24 / 4.35. Grouping leans on object-centric semantics, so the VLM lead carries over.
  • 3D geometry (DL3DV): VGM wins on every metric: camera AUC@30 0.527 vs 0.330, depth AbsRel 0.072 vs 0.113, point-map error 0.152 vs 0.223 (lower is better on the last two).
  • Naive fusion (WAN2.1-T2V-14B + Qwen3-VL-8B, concatenated normalized features): 92.30 mAP, 23.70 T-mIoU, 0.042 depth AbsRel, 0.615 camera AUC@30. It beats both single backbones on semantics and geometry at once.
  • Probe depth is stable: across probe depths 1 to 6 the relative model ranking barely moves, evidence that the probe is reading existing structure rather than learning the task from scratch.
  • Grouping needs semantics, not just continuity: cross-view grouping benefits more from object-centric features than from raw geometric continuity, which is why video models trail despite their geometry edge.

Limits and open questions

This is a probing study, so read it as a measurement of frozen representations, not a recipe for the best spatial model. A full fine-tune could reshuffle the ranking; the paper deliberately does not test that. The benchmarks are narrow: ScanNet20 indoor scenes plus DL3DV, with no outdoor, dynamic, or egocentric stress tests, so “spatial intelligence” here is mostly static indoor 3D. The fusion result is the most exciting and the least examined. It is a single concat of one VGM and one VLM with no scan over which pairs fuse best, no analysis of whether the gains hold at larger scale, and no end-task (navigation, manipulation) to confirm the probe metrics translate. For how these representations feed an actual world model, the world-models topic page collects the systems that would consume this kind of backbone.

FAQ

Vision-language vs video generation for spatial intelligence: which is better?

It depends on the spatial task, and that split is the paper’s main result. Vision-language models win the semantic axis (92.08 vs 69.89 mAP on tagging) and instance grouping (22.66 vs 13.24 T-mIoU), because contrastive image-text training encodes object identity. Video-generation models win 3D geometry (0.527 vs 0.330 camera AUC, 0.072 vs 0.113 depth error), because next-frame prediction forces them to model motion and parallax. The combined answer is that a naive fusion of one of each beats both, so the honest recommendation is to use them together rather than pick a winner.

Do video generation models beat VLMs at depth estimation in this comparison?

Yes, on this benchmark. With both backbones frozen and only a probe trained, video models hit 0.072 average depth AbsRel on DL3DV against 0.113 for VLMs, and 0.527 camera AUC@30 against 0.330. The reason is the training objective: predicting future frames requires an internal sense of depth, occlusion, and camera motion, while a captioning objective never has to. WAN2.1-T2V-14B is the strongest geometry backbone in the study.

Why does instance grouping favor VLMs rather than video models?

Grouping pixels into the same object across views sounds geometric, but the paper finds it leans more on object-centric semantics than on geometric continuity. A VLM already knows “these pixels are a chair,” which is exactly the prior that holds an instance together across viewpoints. Video models track motion well but carry weaker object identity, so they trail on T-mIoU (13.24 vs 22.66) despite leading on raw geometry.

Does naive fusion of a VLM and a video model beat both in spatial intelligence?

In this study, yes. Concatenating normalized features from WAN2.1-T2V-14B and Qwen3-VL-8B and training the same probe yields 92.30 mAP (above the best VLM), 0.042 depth AbsRel, and 0.615 camera AUC@30 (above the best video model). The two paradigms encode complementary information, so a simple concat preserves the VLM’s semantics while inheriting the video model’s geometry. It is one pairing without a search over alternatives, so treat it as a strong existence proof rather than a tuned optimum.

One line: VLMs read meaning, video models read space, and gluing one of each together beats both at once. Read the original paper on arXiv.