World Models · Video Generation · Robotics
Cosmos 3 vs OmniDreams: Two NVIDIA World Model Bets for Physical AI
Cosmos 3 is one open model spanning language, image, video, audio, and action; OmniDreams is a Cosmos-based driving simulator that hits 68 FPS on one GB300 and cut closed-loop incidents from 10.1% to 4.7%.
The question both papers answer
Within two days in June 2026, NVIDIA shipped two world models with opposite design bets. Cosmos 3 is the generalist: one open model that reads and writes language, image, video, audio, and robot action inside a single mixture-of-transformers architecture. OmniDreams is the specialist: a generative driving simulator distilled from the earlier Cosmos-Predict 2.5 diffusion model until it runs a closed-loop AV stack at frame rate.
Both call themselves world models for Physical AI, and both are right. The difference is what each one sacrifices. Cosmos 3 gives up per-task focus to cover five modalities with one checkpoint; OmniDreams gives up breadth, camera count, and openness to hit 68 FPS on a single GPU. Reading them together answers the practical question: when someone says “just use a world model,” which of the two meanings do they actually need?
Cosmos 3: one checkpoint, five modalities
Cosmos 3 routes language, vision, audio, and action tokens through distinct transformer experts that share a joint sequence, so any output can be conditioned on any combination of inputs. Text-to-image, image-to-video, video-plus-instruction-to-action, and world simulation become different routings through the same network rather than different products. The action modality is what makes it a policy model at all: given pixels and an instruction, it emits the next moves, which is why it can appear on a robotics leaderboard.
NVIDIA’s reported results at release: best open model for text-to-image and image-to-video on Artificial Analysis, and the top policy model on RoboArena. The release is unusually open even by 2026 standards: checkpoints, training data, and evaluation benchmarks under the OpenMDW-1.1 license, not just weights. The 294-author author list signals an org-scale engineering effort.
The honest read of the abstract-first evidence: it leads with leaderboard rankings, not parameter counts or ablations, so the exact margins and model sizes are not visible from the paper summary alone, and “best open model” standings shift monthly as competitors ship. A five-modality backbone also costs more to run than a specialist when the task only needs one modality.
OmniDreams: a Cosmos compressed into a real-time driving simulator
OmniDreams does not compete with Cosmos 3; it extends the same lineage downward into a number the robotics world actually needs: frames per second. The base is Cosmos-Predict 2.5, a pretrained diffusion video model. OmniDreams mid-trains it on 21k hours of real driving logs (the RDS dataset: 3M 20-second clips, plus a curated RDS-HQ-1M set of 1.14M clips across 15 countries), then applies Diffusion Forcing with causal masking so each frame attends only to past frames, distills with Self Forcing and Distribution Matching Distillation down to a 2-step generator, and keeps a fixed-size rolling KV cache for arbitrarily long rollouts.
Why bother, when reconstruction-based simulators like NuRec already render captured drives sharply? Because a Gaussian-splatting reconstruction cannot extrapolate: steer off the recorded trajectory, insert an object that was never there, or summon rare weather, and it degrades or refuses. A generative simulator synthesizes those cases, at the price of drift over long rollouts.
The headline numbers, all from the paper:
- Real time: the single-view 2B model renders an 8-frame chunk in 118 ms on one GB300, about 68 effective FPS at 704x1280. The four-view model reaches 105 FPS per camera only on a 16-GPU GB300 cabinet; on a single GPU the same four-view chunk takes 1,289 ms, about 12 FPS.
- Quality: the distilled Self-Forcing model hits FVD 24.8 on RDS-HQ-1M, below its bidirectional teacher’s 26.8 and the many-step causal model’s 31.7, and leads on downstream 3D detection (LET-AP 0.400) and lane-line F1 (0.828) computed on its synthesized frames.
- Long rollouts: a progressive long-context teacher cuts mean segmented FVD over 20 seconds from 240.0 to 179.4, but quality still falls across the rollout: 95.5 in the first five seconds versus 268.4 in the last.
- Closed-loop sim swap (501 scenes): replacing only the sensor simulator inside the AlpaSim stack, the OmniDreams-derived setup logs 4.7% All Incidents versus 10.1% for full Alpamayo 1.5, 20.9% for a 2-camera variant, and 51.9% for a 1-camera variant.
- As a policy (574 scenes): a World-Action Model post-trained from OmniDreams cuts collision rate from 6.9% to 4.2% versus Alpamayo 1.5, with about 2B parameters against roughly 10B. The authors themselves label this preliminary.
Key numbers side by side
| Dimension | Cosmos 3 | OmniDreams |
|---|---|---|
| Role | General omnimodal world model for Physical AI | Specialized generative simulator for autonomous driving |
| Architecture | Mixture of transformer experts over five shared-modality token streams | Cosmos-Predict 2.5 diffusion, causal-masked, distilled to a 2-step generator |
| Modalities | Language, image, video, audio, action | Multi-camera driving video conditioned on text prompt, scenario map, and driving action |
| Real-time goal | Not the point; generation plus policy ranking | The whole point: 68 FPS single-view on one GB300; 105 FPS four-view on a 16-GPU cabinet; 12 FPS four-view on one GPU |
| Generation quality | Best open T2I / I2V on Artificial Analysis at release (NVIDIA-reported) | FVD 24.8 distilled vs 26.8 teacher vs 31.7 many-step causal; LET-AP 0.400; lane-line F1 0.828 |
| Long-horizon behavior | Not reported in the abstract | Segmented FVD 240.0 to 179.4 with progressive teacher; 95.5 first 5s vs 268.4 last 5s |
| Closed-loop result | Top policy on RoboArena at release (NVIDIA-reported) | All Incidents 4.7% vs 10.1% Alpamayo 1.5 in a sensor-simulator swap; WAM collision 6.9% to 4.2% |
| Training data | Not itemized in the abstract; weights, data, and benchmarks released | 21k hours driving logs; 3M clips plus 1.14M curated RDS-HQ-1M clips across 15 countries |
| Openness | OpenMDW-1.1: checkpoints, data, benchmarks | No public code or weights; depends on Cosmos access and NVIDIA-internal AlpaSim/Alpamayo |
| Hardware story | One backbone for perception, generation, and action | 68 FPS needs one GB300; multi-camera frame rate assumes a 16-GPU GB300 cabinet |
One caveat per row where the ruler matters: the OmniDreams quality and closed-loop numbers are all internal same-harness comparisons, and the closed-loop test throttles the policy to the simulator’s 533 ms chunk rate rather than running a full 10 Hz policy. The Cosmos 3 rankings are release-date standings against open competitors, a moving target.
Where the two papers agree
Both reject the idea that a world model is a prettier video generator. Cosmos 3 spends its action modality on robot policy; OmniDreams spends its entire architecture budget on causality so a driving policy can step through it. In both, the world model exists to predict the consequences of an action, which is the definition that separates it from a multimodal chat model: the contrast drawn more broadly in World Models vs LLMs.
Both also admit the same weakness in different units. Cosmos 3’s abstract offers rankings without ablations, so the per-task cost of unification is invisible; OmniDreams measures its own drift directly and still ends a 20-second rollout far worse per window than it began. Long-horizon reliability is the unpaid bill in both papers.
When to use which
Pick Cosmos 3 when the problem is a general Physical AI backbone: a robot stack that needs perception, video generation, and action from one set of weights, with open data and benchmarks to build on. Its value is consistency across modalities and the openness of the release, not topping every specialist leaderboard.
Pick OmniDreams when the problem is AV closed-loop simulation and the existing reconstruction-based simulator hits the extrapolation wall: off-trajectory steering, injected rare events, unseen weather. There, generative simulation is now a working option rather than a demo, provided the hardware budget is a GB300-class cabinet for multi-camera frame rate.
If neither hardware profile fits, Kairos is the third point on the spectrum: a 4B native world model stack that generates a 480P rollout in 11.4 seconds on a single RTX5090 and leads PAI-Bench at 80.84. It is smaller and narrower than Cosmos 3, but it is the option a small team can actually run and modify.
Do not confuse the two FPS figures with each other, and do not quote the 105 FPS number without its 16-GPU cabinet attached; on one GPU the four-view model runs at 12 FPS. Treat the World-Action Model result, a 2B model beating a roughly 10B VLA on collision rate, the way the authors do: one preliminary held-out sweep, not a finished driving stack.
Limits and open questions
Cosmos 3’s limits are evidence-shaped: without parameter counts or ablations in the abstract, the price of five-modality unification cannot be priced from the paper alone, and leaderboard standings depreciate quickly. OmniDreams’ limits are structural: no released code or weights, long-rollout drift that no teacher fully removes, and a closed-loop evaluation that uses NuRec replay as the stand-in for real-world behavior, which weakens as rollouts diverge from the captured trajectory. Whether a World-Action Model trained this way generalizes beyond the NuRec evaluation set is explicitly open.
The open question the pair poses together: if distilling a Cosmos into a causal 2-step simulator buys 68 FPS at FVD 24.8 for driving, what does the same treatment buy for the other four modalities in Cosmos 3? Nobody has published that number yet.
FAQ
Is OmniDreams the same model as Cosmos 3?
No. OmniDreams is built from Cosmos-Predict 2.5, an earlier Cosmos diffusion video model, specialized for driving. Cosmos 3 is a separate, newer omnimodal release covering language, image, video, audio, and action in one mixture-of-transformers model. They share the Cosmos lineage and the Physical AI goal, but one is a generalist and the other is a real-time driving simulator.
Which one is open source?
Cosmos 3, and unusually so: NVIDIA released checkpoints, training data, and evaluation benchmarks under OpenMDW-1.1. OmniDreams has no public code or released weights; reproducing it depends on Cosmos access and the partially internal AlpaSim and Alpamayo stack.
Can OmniDreams really run in real time on one GPU?
For a single camera, yes: the 2B single-view model produces an 8-frame chunk in 118 ms on one GB300, about 68 effective FPS at 704x1280. The four-camera figure needs a 16-GPU GB300 cabinet to reach 105 FPS per camera; the same four-view model manages only about 12 FPS on a single GPU.
What is the World-Action Model result and how much should I trust it?
Post-trained from OmniDreams, a roughly 2B World-Action Model cut collision rate from 6.9% to 4.2% versus the roughly 10B Alpamayo 1.5 on 574 held-out scenes. The authors label it preliminary: it excludes scenes OmniDreams trained on and uses NuRec replay as the real-world reference. Read it as evidence that one generative backbone could both render and drive, not as a validated replacement for a production driving policy.