World Models · Video Generation · Agent Memory
Echo-Memory vs Mirage: What Should a Video World Model Remember?
Echo-Memory isolates the memory module: block-wise recurrence hits 69.0 revisit consistency vs 12.25 with no memory. Mirage moves memory to latent space: 10.57x faster, 55x less GPU memory.
The shared problem: the room has to still be there
An interactive video world model rolls forward in chunks. You give it a start frame, a text prompt and camera actions; it generates video; the camera pans away, explores, and eventually swings back to where it started. A model with no memory happily invents a brand-new room. Everything in this comparison is a different answer to one question: what do you store, and how do you read it back, so the return view matches what was there before?
Echo-Memory and Mirage attack the same problem from opposite directions, and the pairing is what makes them worth reading together. Echo-Memory is a controlled study: freeze the backbone, the action encoding, the data and the 55k-step training budget on 8 A100-80G GPUs, and swap only the memory module, so score differences are attributable to memory design. Mirage is a system: take a released backbone (Wan2.2-TI2V-5B) and rebuild where the 3D memory lives so that an expensive per-chunk round trip disappears. One tells you which family of mechanisms wins; the other shows what winning looks like when engineered into a latent-space pipeline.
Where the memory lives
The design axis both papers expose is the same one: pixel space versus latent space, and summarization versus retention.
Mirage’s baseline, the dominant prior approach, is an explicit 3D point cloud in RGB space. Every chunk, you decode latents to pixels, lift pixels into 3D using depth, render the remembered geometry from the new camera pose, and re-encode the rendered view back into latents. That loop is the bottleneck twice: it is slow, and it is lossy, because the diffusion model’s latent tokens carry learned features that flatten the moment you collapse them to RGB. Mirage’s fix is to never leave latent space: latent tokens are lifted into 3D by depth-guided back-projection into a persistent latent point cloud, and a new view is produced by warping those stored latents directly to the target pose and feeding them to the diffusion model as conditioning. No decode, no render, no re-encode. Because the Wan2.2 VAE has a spatial stride of 16, temporal stride of 4 and 48 channels, each latent token already summarizes a 16x16 pixel block, which is exactly why keeping memory at that resolution is cheaper than per-pixel clouds. One honest choice does heavy lifting: a dynamic-region filter keeps moving objects out of the persistent cache, so only rigid scene structure is remembered.
Echo-Memory’s space of options is broader because its goal is comparison, not a single system. On a shared video-DiT backbone with a per-frame VAE context representation, it tests four memory families: raw context (feed back the last K stored frames, at K=1, 5, 20), compression-based memory (weight-only summaries, length reduction at r=2 and r=4, and hybrids), a learned spatial-memory summary with several read-out paths, and state-space recurrence in both a legacy hybrid form and a block-wise form. Nothing here avoids latent space the way RGB point clouds do; the interesting finding is which of these latent-side designs actually preserves the scene.
Key numbers
| Measurement | Echo-Memory | Mirage | Setting and source | Same harness? |
|---|---|---|---|---|
| Headline revisit-consistency score | 69.0 (block-wise state-space recurrence) vs 12.25 (no-memory I2V baseline) | WorldScore average 70.36 vs prior best Spatia 69.73 | Echo-Memory open-domain VLM score 0–100; Mirage WorldScore 0–100 | No, different metrics and judges |
| Memory read-out ablation | spatial memory 6.00 default → 15.50 (inject-none) → 17.12 (cross-attention read-out) | n/a | Echo-Memory read-out study | Yes, within Echo-Memory |
| Raw-context scaling | 50.75 at K=5 → 58.63 at K=20 | n/a | Echo-Memory raw context family | Yes, within Echo-Memory |
| Compression fragility | weight-only 22.38; length-reduction r=4 43.25; hybrid r=4 collapsed to 9.00 | n/a | Echo-Memory compression family | Yes, within Echo-Memory |
| Recurrence design change | legacy hybrid 34.75 vs block-wise 69.00 | n/a | Echo-Memory state-space family | Yes, within Echo-Memory |
| Revisit geometry metrics | replay PSNR 13.60 (best: spatial memory) | 3D Consistency 92.21; Photometric Consistency 93.95 | Echo-Memory replay regime; Mirage WorldScore components | No |
| Static vs dynamic scenes | n/a | Static Score 73.60 vs Dynamic Score 67.11 | Mirage WorldScore components | Yes, within Mirage |
| Efficiency vs RGB point-cloud memory | n/a | 10.57x faster end-to-end; 55x less GPU memory | Mirage vs explicit RGB point-cloud baselines | Yes, within Mirage |
| Best absolute result | 69.0/100 revisit score | 70.36/100 WorldScore average | Each paper’s own benchmark | No |
Read the table for what it does not contain first. The two headline scores, 69.0 and 70.36, look comparable, since both are 0–100 consistency scores on video world models published two weeks apart, but they are not the same metric. Echo-Memory’s number is a Qwen3-VL judge scoring appearance, presence, view and scene agreement (weighted 0.45/0.25/0.20/0.10) on free camera trajectories with no ground-truth frame; Mirage’s is the WorldScore benchmark average on its standard suite. A model scoring 69 on one would not necessarily score 69 on the other. What is legitimately comparable across the papers is the design logic and the internal ablations, each marked as within-paper.
The two findings that should change how you build
First, from Echo-Memory: read-out matters more than storage, and the clever-looking designs lost. The learned spatial-memory summary scored 6.00, worse than remembering nothing at all (12.25). The stored representation was fine; the default read-out path was destroying the information. Switching read-out to inject-none lifted the same memory to 15.50, and a dedicated cross-attention read-out to 17.12. Meanwhile compression, the family that should have been the efficiency winner, was fragile rather than graceful: weight-only summaries reached 22.38, aggressive length reduction actually helped (43.25 at r=4), but the hybrid of the two collapsed to 9.00. And the winning block-wise recurrence scored 69.00 against 34.75 for a legacy hybrid recurrence, nearly a 2x swing from a structural change inside one family. The lesson generalizes beyond this paper: when a memory mechanism underperforms, ablate the read path before blaming the representation.
Second, from Mirage: the pixel round trip, not the memory itself, was the cost. Keeping the 3D cache in latent space removes a VAE decode, a render and a VAE encode from every chunk, and the payoff is 10.57x end-to-end speed and 55x lower GPU memory at equal-or-better quality. The quality side is deliberately framed by the authors as incremental: 70.36 versus Spatia’s 69.73 is a 0.63-point edge, which on its own would be a minor result. The reason the paper matters is that it makes long-horizon interactive rollouts, the game-like environments and embodied simulators where a scene must persist for minutes rather than seconds, practical on one GPU class of hardware instead of a render farm. The Wan2.2 VAE’s 16x spatial stride is load-bearing: on a backbone with a finer latent grid the same trick buys less.
Where the two papers disagree with each other
They do not run the same experiment, but they point at different conclusions about what to store. Echo-Memory finds that a compressed or summarized spatial representation is the most dangerous thing you can do: compression collapsed, and the spatial summary underperformed no memory until its read-out was fixed. Its winner, block-wise state-space recurrence, is a retained-but-compacted signal carried forward in the model’s own state, not an external cache. Mirage, in contrast, stores an explicit external 3D cache and wins on efficiency, but it stores raw-ish latent tokens, not a learned summary, and it gets to be lossless about it precisely because latent tokens at stride 16 are already a compact representation. In other words, both papers agree that learned summarization of scene memory is where designs go to die; they disagree on whether the answer is recurrence-in-state (Echo-Memory) or an external warped cache (Mirage), and that disagreement is unresolved by any shared benchmark.
There is also a measurement lesson from Echo-Memory. It separates replay (re-run the ground-truth trajectory, diff frames with PSNR/SSIM/LPIPS) from open-domain return (free trajectory, VLM judge). The spatial-memory design that posted the best replay PSNR, 13.60, was the same one that scored worst on semantic revisit, 6.00. PSNR rewards copying; it does not measure whether the model remembers. Mirage’s 3D Consistency (92.21) and Photometric Consistency (93.95) are closer to what matters, but even those are computed within its own benchmark harness. If you adopt either method, budget for a revisit-specific evaluation, not just generation-quality metrics.
When to use which
- Choosing a memory module for an action-conditioned video world model under a fixed training budget: Echo-Memory’s ranking is the best controlled evidence available: block-wise state-space recurrence (69.0), then raw context at K=20 (58.63), then length-reduction compression (43.25), with learned spatial summaries and hybrid compression behind even the no-memory baseline. Read-out design is the first thing to ablate.
- Shipping a long-horizon interactive world model on existing hardware: Mirage is the adoptable system. It wraps a released Wan2.2 checkpoint, needs no retraining of the backbone, and the 10.57x/55x efficiency numbers are against explicit point-cloud pipelines you would otherwise be running.
- Scenes with lots of motion: Mirage’s dynamic-region filter excludes moving objects from persistent memory, and its own numbers show the cost: Dynamic Score 67.11 against Static Score 73.60. A character who walks out of frame is not guaranteed to come back consistent. Echo-Memory does not solve this either; its best score is 69/100, so even the winning mechanism leaves a third of the consistency gap open.
- Evaluating your own system: do not trust replay PSNR alone; Echo-Memory shows a design can win pixel metrics while failing semantic revisit. A VLM-judged open-domain return score, with a judge-swap cross-check (Echo-Memory reports agreement within 1.3–3.1 points and rank correlation 0.93 or better), is the more honest regime.
Limits and open questions
Neither paper settles the question in the other’s setting. Echo-Memory’s rankings come from a single rig (352x640, 81-frame segments, 55k steps, one backbone family) and may not survive higher resolution, longer horizons, or the much larger training budgets where compression methods could finally earn their keep. It is also a study, not a release: you get a ranking and an evaluation protocol, not a tuned checkpoint. Mirage’s results are tied to one backbone with one VAE geometry; whether latent spatial memory transfers to backbones with different strides and channel counts is untested, and its depth-guided back-projection inherits whatever error the depth estimator has. The 0.63-point WorldScore lead over Spatia means the quality claim is an efficiency claim. And the cross-paper gap remains: nobody has run Mirage’s latent cache against Echo-Memory’s recurrence under one harness, so “external latent cache versus in-state recurrence” is still an open design decision with no shared-benchmark answer.
FAQ
Which memory mechanism is best for a video world model?
Under Echo-Memory’s controlled comparison, block-wise state-space recurrence scored 69.0 on open-domain revisit consistency versus 12.25 for no memory, beating raw context at K=20 (58.63) and all compression and spatial-summary designs. For a production system, Mirage’s latent spatial memory is the most practical today because it needs no backbone retraining and runs 10.57x faster than RGB point-cloud pipelines.
Why did Echo-Memory’s spatial memory score worse than no memory at all?
The read-out path, not the stored representation, was the bottleneck. The default spatial-memory read-out scored 6.00, below the 12.25 no-memory baseline; switching to an inject-none read-out lifted it to 15.50 and a dedicated cross-attention read-out to 17.12.
How much faster is Mirage than point-cloud world models?
Mirage runs end-to-end generation 10.57x faster and uses 55x less GPU memory than explicit RGB point-cloud baselines, while reaching a higher WorldScore average (70.36 vs 69.73 for the prior best, Spatia).
Can I compare Echo-Memory’s 69.0 with Mirage’s 70.36 directly?
No. Echo-Memory’s 69.0 is a Qwen3-VL judge score on free-trajectory revisit with no ground truth, while Mirage’s 70.36 is a WorldScore benchmark average. Both are 0–100 consistency scores, but they are different metrics on different harnesses, and the papers’ backbones, resolutions and evaluation protocols differ as well.