Diffusion Language Models · Language Models · Sequence Modeling
Diffusion vs Autoregressive Language Models: What Parallel Decoding Costs
An 8B masked-diffusion LLM matches an 8B autoregressive one on MMLU, beats it on GSM8K with a sixth of the tokens, and fixes the reversal curse. Decoding speed and serving maturity still favor autoregression.
Two ways to factor a sentence
An autoregressive model writes the joint probability of a sequence as a chain of next-token conditionals and generates left to right, one token per forward pass, caching keys and values so each step costs about the same. Llama 3 is the reference point for that recipe at scale: a dense 405B Transformer trained on more than 15T tokens with a 128K context, plus 8B and 70B siblings.
A masked diffusion language model never picks a direction. Training corrupts a sequence by masking each token independently at a random ratio, and a bidirectional Transformer learns to predict all masked tokens at once. Generation starts from a fully masked canvas and runs the process backward over a fixed number of steps: predict, keep the confident tokens, re-mask the rest, repeat. SEDD gave this family a principled loss in 2023 by extending score matching to discrete data, an independent reproduction study confirmed its perplexity and latency claims a year later, and LLaDA scaled it to 8B parameters and 2.3T tokens in 2025, the first fair fight against a same-size autoregressive model.
The two factorizations imply different failure modes. Autoregression pays per token but conditions each token on everything before it. Parallel unmasking pays per step but, in the standard formulation, predicts the tokens in a step independently of each other, which is the factorization error that FeF-DLLM targets.
Key numbers
| Measurement | Autoregressive | Diffusion LM | Setting and source | Same harness? |
|---|---|---|---|---|
| GSM8K, 8B base, 4-shot | 53.1 (LLaMA3 8B) | 70.7 (LLaDA 8B) | LLaDA paper | Yes |
| MMLU, 8B base, 5-shot | 65.4 | 65.9 | LLaDA paper | Yes |
| Reversal poem completion | 34.3 (GPT-4o) | 42.4 (LLaDA 8B Instruct) | LLaDA paper | Yes, same prompts |
| Pretraining tokens to reach that level | 15T (LLaMA3) | 2.3T, 0.13M H800 hours | LLaDA paper, reported budgets | No, different labs |
| Instruction-tuned chat benchmarks | LLaMA3 8B Instruct ahead | LLaDA 8B Instruct trails by small gaps | LLaDA paper | Yes |
| Perplexity vs prior diffusion LMs | n/a | 25 to 75% lower | SEDD paper | Within diffusion family |
| Perplexity vs GPT-2 | GPT-2 baseline | competitive, outperforms GPT-2 | SEDD paper, matched sizes | Yes |
| Generative perplexity without temperature tricks | un-annealed GPT-2 | 6 to 8x better | SEDD paper | Yes |
| Compute to match quality | 1x network evaluations | 32x fewer | SEDD paper | Yes |
| Inference latency | GPT-2 baseline | up to 4.5x more efficient | SEDD reproduction study | Yes |
| HellaSwag, ARC, WinoGrande | GPT-2 class | roughly matched | SEDD reproduction study | Yes |
| Conditional generation from short prompts | stronger | slightly weaker | SEDD reproduction study | Yes |
| Fixing factorization error | n/a | +5.04 points average, 3.86x speedup | FeF-DLLM on GSM8K, MATH, HumanEval, MBPP | Within diffusion family |
| Unique experts touched per decoding block, MoE | per-token routing | 69.5 → 14.6 with block routing, 99.11% accuracy kept | dMoE on LLaDA2.0-mini | Within diffusion family |
The first block of rows is the only 8B-scale head-to-head with a shared recipe, and it comes from the diffusion side’s own paper. The SEDD rows are GPT-2 scale. The last two rows compare diffusion decoders with themselves and are here because they show where the engineering effort is going.
Where diffusion wins
Direction is the cleanest structural win. Because masking is symmetric, a diffusion LM has no built-in left-to-right bias, so tasks that ask the model to complete a sequence backward do not get structurally harder. LLaDA’s 42.4% against GPT-4o’s 34.3% on reversal poem completion is the number to quote; it is a small benchmark, but it is a qualitative capability that autoregressive models inherit as a weakness by construction. The same property gives free infilling and conditioning on tokens at arbitrary positions, which SEDD demonstrated at GPT-2 scale.
Sample efficiency is the surprising win. LLaDA reached LLaMA3-8B-level MMLU and a 17-point GSM8K lead on 2.3T tokens against a reported 15T for LLaMA3. The training corpora differ, so treat this as suggestive rather than a controlled result, but it is the opposite of what most people expected from a non-autoregressive objective.
Parallelism is the promised win and the least settled. SEDD reaches similar quality with 32x fewer network evaluations than autoregressive sampling, and the reproduction study measures up to 4.5x lower latency than GPT-2. At 8B scale LLaDA’s own paper is more cautious: quality trades off against step count and there is no mature analog of the KV cache, so a tuned autoregressive serving stack can still be cheaper per token.
Where autoregression still wins
Maturity. Years of RLHF, speculative decoding, paged KV caches and quantization all assume left-to-right generation. LLaDA 8B Instruct trails LLaMA3 8B Instruct on several chat benchmarks, and the paper attributes it to the post-training recipe being younger, not to the objective.
Conditioning. The reproduction study finds SEDD slightly weaker than GPT-2 when generating from short prompts, which is the common case for chat.
Coherence within a step. Standard diffusion decoders predict the tokens of a step independently, which is why FeF-DLLM’s exact prefix-conditioned factorization gains 5.04 points on average across GSM8K, MATH, HumanEval and MBPP; it recovers that coherence with speculative decoding inside the denoiser and still reports a 3.86x speedup over the sequential fix. Autoregression has this coherence by default.
Serving sparse models. Diffusion decoding unmasks blocks of tokens in parallel, and per-token MoE routing makes a block touch far more experts than a single token would: 69.5 unique experts per block on LLaDA2.0-mini. dMoE’s block-level routing brings that to 14.6 with 99.11% of accuracy kept and a 1.14x to 1.66x speedup. An autoregressive MoE never had the problem.
When to use which
- Chat, agents, code assistants in production now: autoregressive. The tooling gap is larger than the benchmark gap.
- Infilling, editing, constrained generation with anchors on both sides: diffusion LM. This is what the objective is good at, and no autoregressive trick reproduces it cleanly.
- Reasoning tasks where answer order should not matter, or reversal-style retrieval: diffusion LM is worth a trial; the LLaDA GSM8K and reversal results are the evidence.
- Research budgets that care about tokens more than serving cost: diffusion LM. LLaDA’s 2.3T-token result is the strongest sample-efficiency data point in either column.
- Sparse diffusion models in deployment: expect to adopt block-level routing or an equivalent; per-token routing wastes most of the MoE memory savings.
Limits and open questions
The only controlled 8B comparison is in the diffusion paper, against baselines the authors trained themselves. No frontier-scale diffusion LM has been trained, so whether the sample-efficiency advantage survives past 8B is unknown. Latency claims are measured at GPT-2 scale or reported as step-count trade-offs, not as tokens per second on a production server against a speculative-decoding autoregressive stack. And two of the sources here are engineering fixes for problems specific to diffusion decoding, which is a sign of an immature stack as much as a sign of progress.
FAQ
Are diffusion language models better than autoregressive language models?
Not across the board. At 8B, LLaDA matches LLaMA3 on MMLU (65.9 vs 65.4) and leads on GSM8K (70.7 vs 53.1) with fewer training tokens, but its instruction-tuned version trails on chat benchmarks and its decoding stack is less mature.
Is a diffusion LLM faster than an autoregressive LLM?
Depends on scale and setup. At GPT-2 scale SEDD reaches similar quality with 32x fewer network evaluations and up to 4.5x lower latency. At 8B, LLaDA’s quality depends on the number of denoising steps and there is no KV-cache equivalent, so a tuned autoregressive server can still be cheaper per token.
What is the difference between diffusion and autoregressive language models in how they generate text?
Autoregressive models predict one token at a time, left to right, conditioning on the prefix. Diffusion LMs start from a fully masked sequence and unmask many tokens per step in parallel, with no fixed direction, which enables infilling and bidirectional conditioning but introduces factorization error within a step.
Why do diffusion LLMs do better on the reversal curse than autoregressive models?
Because their training objective is symmetric under masking, they carry no left-to-right bias. LLaDA 8B Instruct scores 42.4% on reversal poem completion against 34.3% for GPT-4o.