Language Models · Efficient AI · Reinforcement Learning
Muon vs AdamW: Half the Pretraining Compute, the Wrong Tool After It
Muon reaches AdamW's loss with about half the pretraining FLOPs and now trains DeepSeek-V4, Kimi K3 and Qwen3.8-Next; used for SFT of an AdamW-pretrained model it loses points, and in RL it collapses.
What the two optimizers do to an update
AdamW works element by element. Each parameter keeps a running mean and variance of its gradient, and the update is the mean divided by the square root of the variance, then decayed. Nothing in the rule knows that a weight belongs to a matrix.
Muon works matrix by matrix. It keeps a momentum of the gradient for each 2D weight, orthogonalizes that momentum with a few Newton-Schulz iterations so all its singular values move toward 1, and applies the result. The update therefore has equal energy in every singular direction, which is a spectral whitening. Moonshot’s Muon is Scalable made this usable at LLM scale by adding AdamW-style weight decay, without which weight norms grow past the bf16-accurate range, and by rescaling each update so its root-mean-square sits near 0.2, AdamW’s typical range, so Muon can reuse learning rates and weight decay tuned for AdamW.
The consequence that decides most comparisons: whitening is a strong prior that every direction of a gradient deserves the same step. That is right when gradients are high-rank and clean, as in pretraining, and wrong when they are low-rank or noisy, which is exactly what Rethinking Muon measured in robot-policy and RL fine-tuning.
Key numbers
| Measurement | AdamW | Muon | Setting and source | Same harness? |
|---|---|---|---|---|
| Compute to reach the same loss, compute-optimal | 1x | about 0.5x | 399M to 1.5B models, Moonlight scaling law | Yes |
| 1.2T-token optimizer swap: HumanEval / MATH / GSM8K | 29.3 / 16.1 / 43.8 | 37.2 / 19.8 / 45.0 | Moonlight vs Moonlight-A, identical recipe | Yes |
| Same swap: MMLU / BBH | 60.2 / 45.3 | 60.4 / 43.2 | Same | Yes |
| SFT of an AdamW-pretrained model: MMLU / HumanEval / GSM8K | 71.4 / 79.3 / 89.8 | 70.8 / 77.4 / 85.8 | Qwen2.5-7B on tulu-3, Moonlight paper Table 7 | Yes |
| Pretrain to SFT optimizer pairing, HumanEval | AdamW then AdamW 53.1 | Muon then Muon 57.3; Muon then AdamW 52.4; AdamW then Muon 53.7 | Moonlight Table 6 | Yes |
| Hyperparameters | lr, weight decay, betas | reuses AdamW’s lr and wd after update-RMS matching; adds Newton-Schulz steps and momentum | Design, Moonlight | n/a |
| Parameters covered | all | 2D matrices only; embeddings, output head and MoE router stay on AdamW | Moonlight; Qwen3.8-Next found Muon destabilizes the router | n/a |
| VLA fine-tuning, VLANeXt LIBERO average | 79.45 | 93.65 (Pion 96.35) | Pion paper Table 1 | Yes |
| VLA-Adapter LIBERO Object at 1,500 steps | 32.2% | 97.0% (Pion 100%) | Pion paper Figure 5 | Yes |
| Real robot, pi-0.5, 3 tasks, 20,000 steps | 31.1% | 38.9% (Pion 85.6%) | Pion paper Table 3 | Yes |
| RL with verifiable rewards, GRPO and GMPO on Qwen3-1.7B and 4B | trains, steady gains | collapses, accuracy near zero | Pion paper Figure 6, eight settings | Yes |
| Gradient-norm spikes at constant LR, 25B-A3B stress test | baseline | more spikes with plain Muon; fewer with Muon plus Gated Residual | Qwen3.8-Next Figures 10 and 11 | Yes |
| Frontier adoption for pretraining | historical default | DeepSeek-V4 (1.6T), Kimi K3 (2.8T, per-head variant), Qwen3.8-Next | 2026 reports | n/a |
Rows one to five are controlled swaps inside one paper. The VLA and RL rows are also controlled, on small models. The adoption row is not evidence of a measured gap at frontier scale; the reports cite convergence speed and stability, not an ablation against AdamW.
Where Muon earns its 2x
Pretraining from scratch, 2D weights, high-rank gradients. Moonlight’s scaling law has Muon matching AdamW’s loss at about half the FLOPs across five model sizes, and the controlled 1.2T-token swap shows the gains land on code and math: HumanEval up 8 points, MATH up 3.7, MMLU flat, BBH down 2. The authors find Muon-trained matrices have higher SVD entropy, which they read as the optimizer exploring more directions. Three frontier reports followed. DeepSeek-V4 uses Muon for most matrices at 1.6T parameters with AdamW kept for embeddings, head, mHC gates and norms, plus a guard against exploding attention logits. Kimi K3 orthogonalizes attention projections per head rather than per matrix. Qwen3.8-Next applies Muon to attention, GDN and expert projections but found it destabilized the MoE router early in training and left the router, embeddings and head on AdamW.
The hyperparameter story is a real advantage too. With update RMS pinned to 0.2, Muon takes AdamW’s learning rate and weight decay unchanged, which removes the tuning cost that would otherwise eat into the compute saving.
Where AdamW still wins
Fine-tuning a model that was pretrained with AdamW. Moonlight’s Table 7 is the cleanest evidence: on Qwen2.5-7B, Muon SFT loses 0.6 MMLU, 1.9 HumanEval and 4.0 GSM8K to Adam SFT. The 2x2 ablation explains why: the benefit is tied to the singular-value structure Muon builds during pretraining, and switching optimizers between stages costs points in either direction.
Noisy or low-rank gradients. In vision-language-action training the action head’s gradients are nearly low-rank, and whitening lifts their noise to the same scale as their signal; Muon still beats AdamW here, but only because AdamW converges slowly, and a filtered variant beats both by a wide margin. In RL with verifiable rewards the gradients have low signal-to-noise because rewards are trajectory-level and clipping and importance sampling discard signal; plain Muon collapses across all eight tested settings while AdamW improves steadily. The reverse ablation, a low-pass Muon that also fails, pins the cause on the spectral shaping itself.
Stability under stress. Qwen’s constant-learning-rate stress test shows plain Muon spiking more than AdamW on the same structure, which is why the report pairs Muon with a gated residual that damps the spikes. Muon at scale is a package deal with architecture and precision choices, not a drop-in swap.
When to use which
- Pretraining a dense or MoE model from scratch: Muon on 2D weights, AdamW on embeddings, output head, router and any per-channel or gate parameters. Match update RMS to 0.2 and reuse your AdamW learning rate and weight decay.
- Fine-tuning a checkpoint someone else pretrained with AdamW: AdamW. Muon SFT is slightly worse on the only controlled test.
- Fine-tuning your own Muon-pretrained checkpoint: Muon again; the Muon-then-Muon cell is the best in Moonlight’s 2x2.
- RL with verifiable rewards, GRPO or GMPO: AdamW, or a spectrally filtered variant such as Pion in per-head mode. Never plain Muon.
- Robot policies and other cross-modal heads with low-rank gradients: Muon beats AdamW on convergence speed, but a high-pass variant is the better choice if you can adopt one.
Limits and open questions
The 2x scaling law stops at 1.5B parameters and a 16B-total MoE; frontier adoption is documented but not ablated against AdamW at that scale. The SFT and RL failure results come from 7B, 1.7B and 4B models on narrow task sets. The VLA numbers use two specific policy architectures and one real-robot setup. And no paper runs Muon and AdamW through a full pretrain-to-RL pipeline on the same model, which is the comparison a lab choosing an optimizer for a new model family actually needs.
FAQ
Is Muon better than AdamW?
For pretraining 2D weights from scratch, yes on the evidence: about half the compute to the same loss, with the largest gains on code and math, and adoption by DeepSeek-V4, Kimi K3 and Qwen3.8-Next. For fine-tuning an AdamW-pretrained model or for RL with verifiable rewards, no: Muon loses points in SFT and collapses in RL.
Can Muon reuse AdamW’s learning rate and weight decay?
Yes, once each update is rescaled so its RMS sits near 0.2, matching AdamW’s typical range. Moonshot introduced this so Muon works without a separate hyperparameter search. Weight decay itself must be added, since the original Muon omitted it.
Why does Muon fail in RL fine-tuning where AdamW works?
RL with verifiable rewards produces low signal-to-noise gradients. Muon’s Newton-Schulz step gives every singular direction the same magnitude, so noise directions get as much weight as informative ones and the policy collapses; in the Pion paper accuracy stays near zero across GRPO and GMPO on Qwen3-1.7B and 4B, while AdamW improves steadily.
Which parameters should stay on AdamW when using Muon?
Embeddings, the output head, MoE routers, and per-channel parameters such as norms, biases and gates. Moonlight, DeepSeek-V4 and Qwen3.8-Next all keep these on AdamW; Qwen reports that Muon on the router caused early-training instability.