Multimodal Models · World Models

WBench vs MulTaBench vs VideoKR: Benchmarks That Check the Modality Is Real

Three 2026 multimodal benchmarks attack demos that only look multimodal: WBench runs 1,058 interaction turns, MulTaBench keeps 41% of candidates, VideoKR ships 315K examples.

WBench vs MulTaBench vs VideoKR: Benchmarks That Check the Modality Is Real

The question all three benchmarks ask

A multimodal demo is cheap to fake. A model can score well on a video benchmark by reading text captions, on a tabular-plus-image dataset by ignoring the image, on a world-model leaderboard by producing one beautiful clip. Three benchmarks published within a month of each other in mid-2026 attack that gap from three different directions, and they share one design philosophy: before measuring how good the model is, first make sure the measurement would notice if the model were cheating.

WBench (May 25, 2026) tests interactive video world models by steering them repeatedly instead of scoring one clip. MulTaBench (May 11, 2026) builds a multimodal tabular benchmark by throwing away every dataset where the extra modality is decorative. VideoKR (June 3, 2026) trains and tests video understanding on questions that cannot be answered from text alone. Different modalities, same suspicion.

This guide walks through what each one measures, the numbers that make each credible, and where each one stops being useful.

WBench: can the world survive being steered?

WBench starts from a blunt observation: most video-generation evaluation scores one clip from one prompt, but a world model is supposed to be steered: you give an instruction, watch the result, and react. A single-clip score never tests whether the world holds together across a sequence of interventions.

So WBench is multi-turn. It runs 289 test cases totaling 1,058 interaction turns, roughly 3.7 interactions per case, and it defines four interaction types that cover the ways real users steer a world model: navigation (move through the environment), subject action (make an entity do something), event editing (change what happens), and perspective switching (move the camera). Because the suite mixes text-driven, camera-controlled, and action-conditioned world models, the four interaction types let one benchmark compare systems that are controlled through entirely different channels.

Every turn is decomposed into 22 automatic sub-metrics that roll up into five dimensions: video quality, setting adherence, interaction adherence, consistency, and physical compliance. The credibility move is validation: instead of trusting one large multimodal judge, WBench pairs specialist vision models with multimodal evaluators, and then checks the automatic metrics against human judgment. Every metric reaches a Spearman correlation of at least 0.94 with humans, and four aspects (event editing, subject action, perspective switching, and spatial consistency) hit perfect correlation.

Then it runs 20 state-of-the-art models through this and the headline finding is a non-result: no model leads on all five dimensions. A system can look superb on video quality while violating physics or drifting out of its setting. That is exactly the failure mode single-clip benchmarks hide, and it is why a single leaderboard number for “world model quality” is misleading by construction.

The limits to keep in mind: the automatic metrics inherit the ceiling of the models they are built from, physical compliance is the dimension most likely to be approximated rather than verified, and 289 curated cases are small enough that a model could in principle be tuned toward them. WBench is a protocol more than a permanent ranking.

MulTaBench: is the extra modality actually load-bearing?

MulTaBench’s target is a quieter form of cheating. Plenty of datasets pair a table with a text field or an image, but in many of them the table alone already contains the signal; the image is decoration. Benchmarking multimodal fusion on such datasets measures nothing about fusion.

So before measuring anything, MulTaBench filters. A joint-signal screen requires that using both modalities actually beats using either one alone; roughly 23% of candidate text-tabular datasets failed it. A task-awareness screen drops datasets where a generic frozen embedding already captures everything useful; 36% of the remainder failed that one. Out of 56 unique text-tabular candidates pulled from existing benchmarks, only 41% passed both screens. Image-tabular was harder still, with about 31% acceptance from the literature before manual curation filled out the final set. What survives is 40 datasets (exactly 20 image-tabular and 20 text-tabular), balanced between classification and regression, with row counts from about 400 to 114,000 and 1 to 245 structured columns.

The measurement side is deliberately plain. Two routes turn the unstructured modality into features: a frozen route (e5 for text, DINO-v3 for images, PCA-reduced to about 30 dimensions) and a target-aware route that fine-tunes the encoder on the prediction target first. Then standard tabular learners do the learning: LightGBM, CatBoost, TabM, TabPFNv2, and TabPFN-2.5 drive curation, with XGBoost, RandomForest, RealMLP, TabDPT, TabICLv2, TabSTAR, ConTextTab, and AutoGluon-Multimodal as robustness checks.

The result is consistent enough to be a rule of thumb: target-aware embeddings beat frozen ones across all five primary learners and both modalities, and the effect transfers to learners that were not involved in curation. The sharpest version of the claim: a target-aware variant built on the small encoder outperformed the large frozen encoder. On the PetFinder dataset, LightGBM goes from 77.2% (image only) to 85.7% once both modalities are combined with target-aware representations, and TabPFN-2.5 goes from 81.1% to 88.0%. Scaling the encoder up helps; tuning it to the task matters more.

The candid weakness is structural: the curation pipeline uses tabular models to decide which datasets qualify, so the selection criterion is entangled with the solutions being evaluated: the curation models themselves cannot be fairly scored on MulTaBench, and the target-aware route adds real fine-tuning compute over the frozen one. Read the “tune your embeddings” advice with that asterisk.

VideoKR: can the model answer what only the video knows?

VideoKR attacks the textual-shortcut problem in video understanding. Standard video QA can often be partially answered from captions, titles, or language priors, so a model can look like it understands video while barely watching it. VideoKR’s fix is data design: a human-in-the-loop, skill-oriented pipeline that generates examples targeting progressively deeper video reasoning, with difficulty, diversity, and reliability enforced on both the examples and their chain-of-thought rationales.

The corpus is 315K video reasoning examples over 145K newly collected, CC-licensed, expert-domain videos. Alongside it the authors curate VideoKR-Eval, an expert-annotated benchmark whose questions require genuine video understanding plus knowledge-intensive reasoning, rather than textual shortcuts.

The usage pattern the paper demonstrates is post-training rather than leaderboard-chasing: under a standard SFT→GRPO pipeline, models post-trained on VideoKR outperform prior post-training approaches on knowledge-intensive video reasoning while remaining competitive on general video reasoning, evidence that data design, not just model scale, drives progress on this axis. For builders, the actionable reading is that VideoKR is primarily a training corpus with an anti-shortcut eval attached, not just another leaderboard.

Key numbers

BenchmarkWhat it guards againstScaleNumbers that make it credible
WBenchSingle-clip scores hiding world-model drift289 cases, 1,058 turns, 20 models5 dimensions via 22 sub-metrics; Spearman ≥ 0.94 with humans on every metric, perfect on 4 aspects; no model wins all 5
MulTaBenchDatasets where the extra modality is decorative40 datasets (20 image-tabular + 20 text-tabular)only 41% of 56 candidates passed both screens (23% failed joint-signal, 36% of rest failed task-awareness); TAR beats frozen on all 5 learners; PetFinder 77.2 → 85.7 (LightGBM), 81.1 → 88.0 (TabPFN-2.5)
VideoKRTextual shortcuts in video QA315K examples over 145K CC-licensed videoshuman-in-the-loop skill-oriented pipeline; VideoKR-Eval expert-annotated; SFT→GRPO on VideoKR beats prior post-training on knowledge-intensive video reasoning

Read this table as three one-dimensional instruments, not a ranking. No model is run through all three, no harness is shared, and the metrics (dimension scores, accuracy, post-training transfer) are not convertible into one another. What is comparable is the shared finding: in each case, once the benchmark closes its designated shortcut, apparent capability drops or turns out to be uneven, which is precisely the information a polished demo was hiding.

When to use which

Match the shortcut you most suspect. If you are building or buying interactive video world models (game-engine substrates, robotics simulators, agent environments), WBench is the only one of the three that measures sustained steering, and its per-dimension scorecard (quality, setting, interaction, consistency, physics) tells you which axis a model fails on instead of averaging the failure away. If your work is tabular learning with attached text or images, MulTaBench is both a benchmark and a lesson: run its filter logic over your own datasets before trusting any multimodal-fusion number, and budget for target-aware embeddings, because the small-tuned-beats-large-frozen result says encoder scale is not where the headroom is. If you are training video reasoning models, VideoKR is the one that ships training data (315K examples) rather than just an eval; use VideoKR-Eval as the shortcut detector for whatever you train.

And if you are reading vendor numbers rather than producing them, the cross-benchmark lesson is the buyer’s version of the same point: ask which shortcut each reported score was designed to close. A number produced against a benchmark that never tests interaction, never checks the modality is load-bearing, or never blocks text-only answers is measuring something easier than the claim attached to it.

Limits and open questions

All three benchmarks share one structural limit: each closes a specific shortcut, which means each is blind to the shortcuts it was not built to close. WBench’s automatic metrics inherit the ceiling of the vision and multimodal models they are composed from, physical compliance is approximated rather than verified, and 289 curated cases are small enough to be tuned toward. MulTaBench’s selection criterion is entangled with the tabular models doing the selecting, so those models cannot be fairly scored on it, and the target-aware recipe it recommends adds fine-tuning compute that the headline accuracy numbers do not price in. VideoKR’s benchmark results are measured under one post-training recipe (SFT→GRPO) and one data pipeline, so the 315K-example recipe may not transfer to a different base model or training stack, and the expert-domain video distribution may not cover general web video. None of this sinks the benchmarks; it bounds what each number can honestly claim.

FAQ

What does WBench measure that other video benchmarks do not?

WBench measures interactive behavior. Instead of scoring one clip from one prompt, it runs 1,058 interaction turns across 289 cases and four interaction types (navigation, subject action, event editing, perspective switching), scoring each generation on five dimensions with 22 sub-metrics validated against human judgment.

How does MulTaBench decide a dataset is truly multimodal?

Two screens: a joint-signal test (both modalities must beat either alone) and a task-awareness test (a frozen embedding must not already capture all useful signal). Of 56 text-tabular candidates only 41% passed both; roughly 23% failed joint-signal and 36% of the remainder failed task-awareness.

What is the main finding of MulTaBench?

Target-aware embeddings, meaning encoders fine-tuned on the prediction target, beat frozen pretrained embeddings across all five primary learners and both modalities, and a small tuned encoder can beat a large frozen one. On PetFinder, LightGBM improves from 77.2% (image only) to 85.7% with both modalities plus target-aware representations.

Is VideoKR a benchmark or a training dataset?

Both. The corpus is 315K video reasoning examples over 145K CC-licensed expert-domain videos, and VideoKR-Eval is an expert-annotated benchmark whose questions require genuine video understanding. The paper’s main result is about post-training: SFT→GRPO on VideoKR beats prior post-training approaches on knowledge-intensive video reasoning.

Can I compare scores across WBench, MulTaBench, and VideoKR?

No. The three measure different modalities with different metrics on different models. Compare them at the level of design: each closes a specific shortcut (single-clip scoring, decorative modalities, textual shortcuts), and the right choice depends on which shortcut threatens your use case.

One line: WBench, MulTaBench, and VideoKR all do the same thing to different modalities — they close the cheap shortcut before counting the score. Read the WBench paper, the MulTaBench paper, and the VideoKR paper for the full detail.