Research Papers
EN ZH

Language Models · Efficient AI · Open Models

Muon Is Scalable Explained: 2x Compute Efficiency Over AdamW

Moonshot AI shows Muon, a matrix-orthogonalizing optimizer, reaches AdamW's loss with about half the compute once you add weight decay and rescale its updates; Moonlight, a 16B MoE trained on 5.7T tokens, is the proof.

Moonshot AI · Paper published 2025-02-24 · 6 min read

Muon Is Scalable Explained: 2x Compute Efficiency Over AdamW

Quick answer

Muon replaces AdamW’s per-element normalization with a per-matrix step: it takes the momentum matrix of each weight, orthogonalizes it with a few Newton-Schulz iterations so all singular values move toward 1, and applies that as the update. Keller Jordan’s 2024 version worked on small models. This February 2025 paper from Moonshot AI shows what it takes to run it at LLM scale: add AdamW-style weight decay, and rescale every update so its root-mean-square matches AdamW’s usual 0.2 to 0.4, which lets Muon reuse learning rates and weight decay tuned for AdamW. With those two changes, scaling-law fits show Muon reaching the same loss as AdamW with roughly half the training FLOPs, and the authors release Moonlight, a 3B-activated, 16B-total mixture-of-experts model trained on 5.7T tokens with Muon.

The two fixes that made it scale

The first problem showed up as growth. Without weight decay, Muon-trained weights and layer outputs kept increasing in RMS until they left the range bf16 represents accurately. The original Muon omitted weight decay entirely; the paper adds the standard decoupled decay from AdamW and shows in Figure 2 that Muon without it starts faster but is overtaken later, while Muon with decay stays ahead of AdamW for the whole run of an 800M-parameter model trained on 100B tokens.

The second problem is that an orthogonalized update has an RMS that depends on the matrix shape: roughly the square root of one over the larger dimension. That makes the effective step size vary across layers and differ from AdamW’s, so the two optimizers cannot share hyperparameters. The fix is to multiply each update by 0.2 times the square root of the larger matrix dimension, pinning the update RMS to about 0.2. The paper tests three variants of this idea on the 800M model and reports that both the normalized-update and adjusted-learning-rate versions reach validation loss 2.789 against 2.812 for the unadjusted baseline. The practical consequence is the one people use: Muon can take AdamW’s learning rate and weight decay as is.

Everything that is not a 2D weight matrix, meaning embeddings, the output head and per-channel parameters, stays on AdamW. Muon also needs a distributed implementation that shards momentum ZeRO-1 style and gathers full matrices only for the Newton-Schulz step; the paper releases that code.

Key results

  • Scaling law: across five model sizes from 399M to 1.5B parameters trained at compute-optimal token counts, the fitted curves in Figure 3 show Muon needs about half the FLOPs of AdamW to reach the same loss. The paper states this as approximately 2x computational efficiency.
  • Controlled optimizer swap at 1.2T tokens (Table 4): Moonlight (Muon) against Moonlight-A (identical architecture, data and schedule, AdamW). MMLU 60.4 vs 60.2, HumanEval 37.2 vs 29.3, MBPP 52.9 vs 49.2, GSM8K 45.0 vs 43.8, MATH 19.8 vs 16.1. BBH goes the other way, 43.2 vs 45.3. Both beat DeepSeek-V3-Small, a similar MoE trained on 1.33T tokens with AdamW.
  • Final Moonlight at 5.7T tokens (Table 5): MMLU 70.0, MMLU-Pro 42.4, HumanEval 48.1, MBPP 63.8, GSM8K 77.4, MATH 45.3, with 2.24B activated parameters. Qwen2.5-3B, trained on 18T tokens, scores 65.6 MMLU and 79.1 GSM8K; Llama3.2-3B at 9T tokens scores 54.7 and 34.0.
  • Where the gain concentrates: the authors note Muon’s advantage is largest on math and code, and Table 4 bears that out: HumanEval and MATH move by 8 and 3.7 points while MMLU moves by 0.2.
  • Singular spectrum: SVD entropy of Muon-trained weight matrices is higher than AdamW’s across layers, which the authors read as Muon exploring more directions of the weight space.

Where Muon does not help

Section 3.5 is the part to remember before switching an existing model to Muon. Fine-tuning Qwen2.5-7B, an AdamW-pretrained model, on the tulu-3 SFT mixture gives Adam 71.4 MMLU, 79.3 HumanEval and 89.8 GSM8K against Muon’s 70.8, 77.4 and 85.8 (Table 7). Muon at SFT time on an AdamW-pretrained model is slightly worse, not better. The 2x2 ablation on Moonlight (Table 6) points at the reason: the best cell is Muon pretraining followed by Muon SFT (MMLU 55.7, HumanEval 57.3, GSM8K 68.0), and switching optimizers between stages loses several points in either direction. Muon’s benefit comes from the singular-value structure it builds during pretraining; it is not a drop-in improvement for post-training someone else’s checkpoint. Follow-up work on Muon vs AdamW beyond pretraining, including reinforcement learning and robot policies, found sharper failures of the same kind.

Limits and open questions

The scaling law tops out at 1.5B parameters and the flagship is a 16B-total MoE; the paper does not show the 2x holding at frontier scale, though DeepSeek-V4 and Kimi K3 later adopted Muon for pretraining at 1.6T and 2.8T parameters. The 2x figure comes from fitted curves, and the AdamW baseline’s hyperparameters were themselves searched (Appendix B), so the comparison is fair but model-family specific. Muon adds a Newton-Schulz step per matrix per iteration and a full-matrix gather in distributed training, which the paper reports as manageable but nonzero overhead. And the paper leaves open how to bring the non-matrix parameters, still on AdamW, into the same framework.

This paper is one of the sources behind DeepSeek V4 Architecture Explained, Kimi K3 Architecture Explained, Qwen3.8-Next Architecture Explained.

FAQ

What are the two changes that let Muon scale to LLM training?

Adding AdamW-style weight decay, because without it weight and activation RMS grow past bf16’s accurate range, and rescaling each orthogonalized update by 0.2 times the square root of the larger matrix dimension so the update RMS matches AdamW’s typical 0.2 to 0.4. The second change lets Muon reuse learning rates and weight decay tuned for AdamW.

How much more efficient is Muon than AdamW according to this paper?

Fitted scaling laws over models from 399M to 1.5B parameters show Muon reaching the same loss with about half the training FLOPs, which the paper states as roughly 2x computational efficiency. In the controlled 1.2T-token comparison, HumanEval rose from 29.3 to 37.2 and MATH from 16.1 to 19.8 while MMLU was flat.

Does Muon help when fine-tuning a model pretrained with AdamW?

No. On Qwen2.5-7B, Muon SFT scored 70.8 MMLU and 85.8 GSM8K against Adam’s 71.4 and 89.8. The Moonlight ablation shows the gain requires Muon in pretraining; mixing optimizers across stages loses points either way.

What is Moonlight and how was it trained?

Moonlight is Moonshot AI’s 16B-total, 2.24B-activated mixture-of-experts model trained on 5.7T tokens with Muon. At the end of training it scores 70.0 MMLU, 48.1 HumanEval and 45.3 MATH, ahead of DeepSeek-V2-Lite trained on the same token count with AdamW.