Diffusion Models · Transformers · Text-to-Image
MV-Split vs DAR: Fixing the DiT Residual Stream
Two 2026 papers agree the inherited residual stream was never designed for diffusion: MV-Split cures deep-layer collapse (1000 layers, FID 2.77), DAR cures magnitude inflation (FID 9.67 to 7.56, 8.75x fewer iterations).
The shared problem: a language-model part inside a diffusion model
Every mainstream image and video generator since DiT runs a transformer that was architecturally copy-pasted from NLP. The residual connection x = x + sublayer(x), PreNorm, LayerScale, all of it arrived pre-owned. Two 2026 papers looked at that inherited plumbing, measured it, and found it misbehaves in ways language models never exposed, because diffusion reuses one network across dozens of noise levels and because image DiTs are being pushed to depths language models rarely attempt.
MMS (Mean Mode Screaming) attacks the depth axis: past a few hundred layers, image DiTs quietly stop improving or fall over, and nobody had a named, mechanistic explanation. DAR (Diffusion-Adaptive Routing) attacks the diffusion axis at standard depth: even a 28-block SiT has pathologies a language transformer does not, because no one ever re-examined the residual add for a model run at many noise levels.
Both papers are diagnostic before they are architectural. That is what makes them worth reading together: they independently land on “the residual stream, as inherited, is the wrong object for a diffusion transformer,” and each one fixes a different way it breaks.
For orientation: the denoising formulation both models serve is much older than either fix; it goes back to DDPM, which established the score-based training loop this whole family inherits. What is new here is not the diffusion process but the transformer plumbing inside it.
Failure mode one: the mean takes over as depth grows
MMS’s mechanism is unusually specific for a collapse paper. Attention maps are row-stochastic: each row of the softmax matrix sums to 1. That property preserves the pure token-mean component of a sequence while damping the centered, token-to-token variation. Stack enough layers and the centered component shrinks multiplicatively; token representations homogenize until cosine similarity between tokens in deep layers approaches 1.0. Once tokens are aligned, the softmax Jacobian’s null space wipes out query and key gradients, attention can no longer learn, and the network locks into a mean-dominated state.
Two measurable signatures come with the diagnosis. The collapse is a sharp event, not a slow drift: the residual writer-gradient norm spikes abruptly at the moment of entry, which is the “screaming” in the name. And the author fits an alignment-amplification law predicting that gradient amplification scales with token alignment; attention and FFN writers both fit with R-squared above 0.9, and the amplification at a divergence event reaches roughly 13x.
Standard stabilizers do not fix this because they treat both components identically. LayerScale multiplies the mean-coherent and the centered parts by the same per-channel scalar, so it can damp the runaway mean mode only by also crushing the signal that matters. That is why deep DiTs under LayerScale historically “just diverged” with no clean story.
Failure mode two: the stream inflates forward, starves backward
DAR’s authors measured three pathologies across the 28 blocks of SiT-XL/2, none of which require extreme depth. First, forward magnitude inflation: the additive residual stream grows in norm as you go deeper, and because PreNorm normalizes each block’s input, the deeper blocks effectively get diluted inputs and train weakly. Second, backward gradient decay: the same accumulation that inflates the forward pass starves the backward one, so gradients reaching early layers shrink fast and the bottom of the network under-trains. Third, inter-block redundancy: adjacent blocks end up computing nearly identical updates, wasting the capacity a 28-layer model should spend on different features.
The timestep dimension is what makes this distinctly a diffusion problem. The same network runs at high noise early in sampling and near-clean images late, but a static residual forces identical information routing at every noise level. DAR replaces the add with a learnable, timestep-aware weighted aggregation over the history of previous sublayer outputs. Because the aggregation is non-incremental, a block can down-weight or skip earlier contributions instead of carrying their accumulated magnitude forward. Because the weights are timestep-adaptive, routing can differ between a high-noise step and a near-clean one.
Key numbers
| Measurement | MV-Split (MMS) | DAR | Setting and source | Same harness? |
|---|---|---|---|---|
| Primary goal | Train past a few hundred layers without collapse | Cut FID and training iterations at standard depth | The two papers’ own framings | Different goals |
| Depth demonstrated | 400 and 1000 layers | 28 blocks (SiT-XL/2) | Each paper’s headline setup | No |
| ImageNet 256 FID | 2.60 (400-layer), 2.77 (1000-layer) | 7.56, down from 9.67 baseline | MMS: own DiT, latents, 50k steps; DAR: SiT-XL/2, static config | No, not cross-comparable |
| ImageNet 256 IS | 185.5 (400-layer), 217.3 (1000-layer) | Not reported in our summary of the paper | MMS on latents | No |
| Baseline at same setup | LayerScale 400-layer: FID 2.90, IS 165.5 at the same checkpoint, and it diverged before the schedule finished | SiT-XL/2 plain residual: FID 9.67 | Each paper’s own control | Yes, within each paper |
| Training efficiency | Not the claim; stability is | 8.75x fewer iterations to match the baseline’s converged quality; about 2x early-stage speedup stacked on REPA | DAR, FID-versus-iteration curve | Yes, within the DAR paper |
| Mechanism evidence | Centered gradient band 2-3x higher than LayerScale at equal depth; amplification law fit R-squared > 0.9, ~13x at divergence | Three measured pathologies: forward inflation, backward decay, inter-block redundancy | Each paper’s diagnostics | No shared metric |
| Inference cost | Adds only per-feature gains in the residual path | Static variant: essentially none; dynamic variant adds routing overhead | Papers’ own statements | No shared metric |
The row that matters most is the one saying “not cross-comparable.” MMS’s 2.60 and DAR’s 7.56 are both ImageNet-256 FID numbers, but they come from different backbones, different latent-vs-pixel setups, and different training budgets. Anyone quoting one as beating the other is misreading both papers.
What each paper found that should change how you build
From MMS: depth scaling was the one axis transformers could not push freely, and the reason was a named, mechanistic failure mode, not a vague “optimization got hard.” The fix is cheap and targeted. MV-Split routes the residual’s mean-coherent and centered components through separate learnable per-feature gains: the mean subspace gets a leaky-integrator path that suppresses the runaway mode, the centered subspace keeps a standard residual path that lets token-to-token variation survive. It adds only per-feature gains, and it preserves a centered gradient band 2-3x higher than LayerScale at the same depth. The practical reading: if your DiT roadmap involves hundreds of layers, the blocker is gone; the 1000-layer run is a stability proof, not a quality record, since its FID (2.77) is slightly worse than the 400-layer run (2.60).
From DAR: the converged quality of a standard-depth DiT was being paid for in iterations that a routing fix recovers. The static variant, with fixed learned coefficients and no timestep conditioning, captures the headline FID gain at essentially no inference cost, which is the configuration a practitioner should evaluate first. The 8.75x iteration reduction is measured against the baseline’s final quality on the FID-versus-iteration curve, i.e., it is an early-to-mid-training comparison, not a guarantee that the factor survives to a very long converged run.
Where the two papers agree and diverge
They agree on the premise: the additive residual stream was tuned for language modeling and never re-derived for diffusion, and measuring it exposes real pathologies. They disagree, implicitly, about which pathology is binding. MMS says the centered signal dies with depth, so rescue it with separate gains. DAR says the accumulated magnitude itself is the problem, so stop accumulating and route instead. These are not contradictory fixes; a routing layer could in principle allocate between mean and centered modes the way MV-Split’s gains do. Neither paper tests the combination.
They also differ in scientific posture. MMS leads with a falsifiable law (alignment predicts amplification, fit R-squared above 0.9) and a stability demonstration at an extreme depth where baselines cannot run at all. DAR leads with an engineering number (8.75x fewer iterations) on a standard architecture where the baseline is well-characterized. MMS tells you why deep DiTs died; DAR tells you what the residual was costing you while they lived.
When to use which
Use MV-Split when your bottleneck is depth: video DiTs and high-resolution cascades are exactly the models being stacked toward hundreds of layers, and that is where LayerScale-class stabilizers have been silently failing. Use DAR when your bottleneck is compute at standard depth: if you are pre-training a 20-to-30-block DiT and the FID-versus-iteration curve is your budget, static DAR is a drop-in change to evaluate before reaching for more data or a new sampler. If you are doing neither, both papers still justify an audit: measure token cosine similarity across depth in your own model, and measure forward norm growth and early-layer gradient magnitude. The diagnostic transfers even where the exact remedy does not.
Limits and open questions
MMS’s 1000-layer model does not beat its own 400-layer model on FID, so the depth claim is about stable training, not free quality; whether depth pays off at all at these scales is still an open empirical question. Its ImageNet-256-latent, 50k-step setting is also far from a frontier text-to-image training run, where the data distribution and conditioning change the picture.
DAR’s headline efficiency factor is an early-to-mid-training comparison, and its strongest results are on one architecture family (SiT) with FID as the metric; FID rewards distribution match and can move without a visible perceptual change, so the gain needs confirmation at higher resolution and against human preference. The dynamic, timestep-conditioned variant’s overhead is the detail to scrutinize before adopting it.
For both: the fixes are evaluated inside DiT-style backbones. Whether the same pathologies bind in unified multimodal transformers that interleave image and text tokens is exactly the question the next generation of architectures will answer.
FAQ
Can I compare DAR’s FID 7.56 with MV-Split’s 2.60 directly?
No. The numbers come from different backbones (SiT-XL/2 vs the MMS paper’s own DiT), different input setups (pixels vs latents), and different training budgets. What you can compare are the within-paper deltas: DAR against its own 9.67 baseline, and MV-Split against its own LayerScale control at the same checkpoint.
Why did LayerScale fail where MV-Split succeeded?
LayerScale multiplies the mean-coherent and centered components of the residual by the same per-channel scalar, so damping the runaway mean mode also crushes token-to-token variation. MV-Split gives each component its own gain, suppressing the mean mode while preserving the centered path, which is what keeps the centered gradient band 2-3x higher at equal depth.
What is the cheapest fix to try first on a standard-depth DiT?
DAR’s static variant. It uses fixed learned routing coefficients instead of timestep-conditioned ones, captures the headline FID improvement, and adds essentially no inference cost. The dynamic variant buys timestep adaptivity at the price of routing overhead you should measure before committing.
Does the 1000-layer DiT set a new ImageNet FID record?
No, and the paper is honest about it: the 1000-layer model reaches FID 2.77, slightly worse than the 400-layer MV-Split run at 2.60. The claim is that training to that depth is now stable, not that depth alone produces better images.
Do the two fixes stack?
Nobody has shown it. They diagnose different pathologies in the same residual stream, and a timestep-adaptive router that also separates mean and centered components is a plausible combination, but neither paper tests it. Treat stacking as a research hypothesis, not an established result.