Mixture of Experts · Transformers · Efficient AI · Diffusion Models
MPI vs DAR vs dMoE: Three Routing Redesigns for Transformers, With Numbers
Three 2026 papers rebuild transformer routing: MPI lifts 11B MoE avg accuracy 40.92 to 42.76, DAR cuts SiT-XL/2 FID 9.67 to 7.56 using 8.75x fewer iterations, dMoE shrinks blocks from 69.5 to 14.6 experts.
The same complaint, heard at three levels
None of these three papers set out to build a bigger model. Each one starts from the same observation: the path that decides where information goes inside a transformer was fixed by convention, and the convention no longer matches the workload. The conventions differ by level, so the fixes do too.
Manifold Power Iteration (MPI) works at the expert level. A sparse MoE layer routes each token through a learned router matrix whose rows stand in for experts, but nothing in the standard objective forces a row to encode anything about the expert’s own weights. MPI’s fix is to update each row toward the principal singular direction of its expert’s weight matrix, then retract it onto a scale-controlled constraint so routing balance survives. One power-iteration step per row keeps it cheap: the authors report a 0.2% training slowdown at 11B, with no inference overhead because the transformed router can be precomputed at model load.
Diffusion-Adaptive Routing (DAR) works at the depth level. The residual add x = x + sublayer(x) was inherited from language transformers, and the paper measures three pathologies that get worse across the 28 blocks of SiT-XL/2: the residual stream’s norm climbs monotonically, gradients reaching early layers decay sharply, and adjacent blocks compute near-duplicate updates. DAR keeps a history of sublayer outputs and forms each block’s input as a learned, timestep-adaptive weighted combination of that history. The static variant, fixed learned coefficients, adds almost no inference cost and still delivers the headline gain.
dMoE works at the decoding-block level. Diffusion language models unmask a whole block of tokens in parallel, but MoE routing still picks experts per token, so one block of LLaDA2.0-mini touches 69.5 distinct experts on average and all of them must sit in memory for the parallel step. dMoE sums the token-level routing scores across the block, applies top-p selection to get one shared expert coreset, and restricts every token to that coreset. Tokens still route individually, just within a working set the block as a whole considers important.
Key numbers
All figures below are the authors’ own, and every comparison is a before/after inside a single paper. No two of the three papers evaluate on the same model, task, or metric, so cross-paper ranking is not meaningful.
| Measurement | Before | After | Setting and source | Same harness? |
|---|---|---|---|---|
| Average accuracy, 7 downstream tasks, 11B MoE | 40.92 | 42.76 | MPI midtraining result | Yes, same model |
| GSM8K, same 11B run | 17.89 | 27.60 | MPI paper | Yes, same model |
| Validation bits per byte, 11B | 0.728 | 0.723 | MPI paper | Yes, same model |
| Math-set validation bpb, 11B | 1.852 | 1.581 | MPI paper | Yes, same model |
| Training throughput | 34.97B tokens/day baseline | +0.2% slowdown with MPI | MPI paper, 11B setup | Yes |
| 1B average over 25 benchmarks, four optimizers | 42.26 to 43.01 before | gains of +0.54 to +1.34 after MPI | MPI optimizer sweep (AdamW, AdamH, Muon, MuonH) | Yes |
| Router load balance, MaxVio Global, 3B | 0.964 | 0.711 | MPI paper | Yes |
| ImageNet 256x256 FID, SiT-XL/2 | 9.67 | 7.56 | DAR, static configuration | Yes, same model |
| Training iterations to reach baseline final FID | 1x | 8.75x fewer | DAR paper, early-to-mid training curve | Yes |
| Early-stage training speed stacked on REPA | 1x | about 2x acceleration | DAR paper | Yes |
| Unique experts touched per decoded block, LLaDA2.0-mini | 69.5 | 14.6 | dMoE paper | Yes, same model |
| Accuracy retained after coreset routing | 100% | 99.11% | dMoE on MATH500, GSM8K, ARC-C, MMLU math | Yes, same model |
| Expert-weight memory | 100% | down 76.64% to 79.84% | dMoE paper | Yes, same model |
| End-to-end latency | 1x | 1.14x to 1.66x speedup | dMoE paper, operating-point dependent | Yes |
| Cross-paper comparison | n/a | n/a | three different models, tasks, and metrics | No, never run together |
Two numbers earn their papers their slots. For DAR it is the 8.75x iteration reduction, because DiT pretraining is compute-bound and a drop-in residual-path change that reaches the same FID sooner is adoptable without restructuring a pipeline. For dMoE it is the 76 to 80% memory cut, because it converts MoE diffusion LLMs from a memory-bound curiosity into something that fits tighter serving budgets. MPI’s case rests on the pattern holding from 1B to 3B to 11B and across four optimizers, rather than on any single benchmark.
What is actually being redesigned
The three methods route different things, and the differences decide when each one applies. MPI changes what a router row means: from a free vector to a compressed proxy that tracks the expert’s dominant weight direction. It is a representation fix, applied during training, and it composes with other routing tricks because it leaves the top-k dispatch interface untouched. Its load-balance numbers (MaxVio Global 0.964 to 0.711 at 3B) suggest part of the gain comes from calmer routing, not only from better alignment.
DAR changes what depth means. A residual stream can only accumulate; DAR’s history aggregation lets a block down-weight or skip earlier contributions, which is what attacks the measured magnitude inflation and gradient decay. The timestep-adaptive version adds a second axis, diffusion runs the same network at many noise levels, and static residuals cannot route differently at each. The static variant matters more for practitioners because it is nearly free at inference.
dMoE changes what a decoding step means. Block-parallel decoding wants a small shared working set; per-token routing wants maximal per-token freedom. These pull in opposite directions, and dMoE resolves the conflict at the routing layer by deriving the coreset from the model’s own learned scores, so high-traffic experts survive and one-stray-token long-tail experts get dropped. Notably, this is the only one of the three that is a serving-time method: nothing about the model’s weights changes.
When to use which
Use MPI when you are pretraining or midtraining a sparse language model and can afford a one-line router update in the training loop. The honest caveat is that it is evaluated against vanilla MoE routers; how much of the gain survives combination with the strongest current load-balancing losses is an open question the paper leaves for later work.
Use DAR when you are training a DiT-class image model and FID per training dollar is the metric. Use the static variant first. The 8.75x figure is an early-to-mid-training comparison, so treat it as a reason to start, not as a guaranteed end-to-end speedup at full convergence.
Use dMoE when you already serve a MoE diffusion LLM and memory, not quality, is the binding constraint. The evaluation is a single open model, LLaDA2.0-mini, and a reasoning-heavy benchmark set, so check your own workload’s accuracy retention before trusting the 99.11% figure.
Limits and open questions
All three results are author-run and none has an open reproduction. MPI and DAR are training-time changes whose reported overheads (0.2% slowdown, near-free static inference) are conditional on specific stacks. dMoE is the narrowest: one base model, four benchmarks, and an average accuracy figure that does not break out which tasks lose the most. The larger open question is composability. A modern diffusion LLM is simultaneously a MoE model, a diffusion model, and a deep transformer, which means in principle it could want MPI’s router alignment, DAR’s depth routing, and dMoE’s block coresets at once. Whether the three fixes stack or interfere is not tested anywhere in this set, and it is the experiment worth running next.
For background on the routing mechanisms these papers build on, see Mixture of Experts Explained and the comparison of diffusion versus autoregressive language models.
FAQ
How much does MPI improve an 11B MoE model?
In the paper’s 11B run, average accuracy over seven downstream tasks rises from 40.92 to 42.76, GSM8K rises from 17.89 to 27.60, and validation bits per byte improve from 0.728 to 0.723, all same-model before/after comparisons.
What results does DAR report for SiT-XL/2 on ImageNet?
On ImageNet 256x256, the static DAR configuration lowers FID from 9.67 to 7.56 and reaches the baseline’s converged quality with 8.75x fewer training iterations; stacked on REPA it gives about 2x early-stage acceleration.
How much expert memory does dMoE save on LLaDA2.0-mini?
dMoE cuts expert-weight memory by 76.64% to 79.84% and speeds up end-to-end latency by 1.14x to 1.66x, while retaining 99.11% of the original accuracy across its evaluated benchmarks.
Do the three methods require retraining the model?
MPI updates router rows during training, DAR is trained together with the model, and dMoE is a serving-time routing change that needs no retraining at all.
How do MPI, DAR and dMoE differ in what they route?
MPI routes tokens to experts by redefining what a router row means, DAR routes sublayer outputs across depth by replacing the residual add, and dMoE routes whole decoding blocks to a shared expert coreset.
Which numbers from these papers are safe to compare?
Only the within-paper before/after pairs quoted above. The three papers use different models, tasks, and metrics, so any cross-paper ranking would be invented rather than measured.