Research Papers
EN ZH

Language Models · Open Models · Long Context · AI Agents

Kimi K3 Explained: Delta Attention, Attention Residuals and 896 Experts

Kimi K3 is a 2.8T-parameter MoE with 104B active, native vision and 1M context, built on 3:1 Kimi Delta Attention and gated MLA, attention residuals and a 896-expert LatentMoE, at 2.5x K2's scaling efficiency.

Moonshot AI · Paper published 2026-07-27 · 6 min read

Kimi K3 Explained: Delta Attention, Attention Residuals and 896 Experts

Quick answer

Kimi K3 is Moonshot AI’s July 2026 flagship: a mixture-of-experts model with 2.78T total parameters, 104.2B activated per token, 93 layers, native vision through a from-scratch MoonViT-V2 tower, and a 1M-token training context. Against Kimi K2 (1.04T total, 32.6B active, 61 layers, 128K context) the architecture changes in three places. Token mixing becomes a layerwise hybrid of Kimi Delta Attention (KDA), a gated delta-rule linear attention, and gated Multi-head Latent Attention, three KDA layers per one global-attention layer. Depth mixing adds Attention Residuals, so each layer can retrieve representations from all preceding layers instead of reading one accumulated residual. And the feed-forward side becomes Stable LatentMoE with 896 routed experts, 16 active per token and 2 shared, routed through a half-width latent. The report attributes an approximately 2.5x improvement in overall scaling efficiency over K2 to these plus data and recipe changes. Weights are open, and at release K3 was the first open model to top the WebDev Arena.

Architecture at a glance

Hybrid attention, 3:1. Each block is three KDA layers followed by one gated MLA layer, with an extra gated MLA layer at the end so the final layer is always global attention. KDA extends the delta-rule recurrence with a channel-wise forget gate, which lets a fixed-size state overwrite stale content rather than accumulate it. Two changes from Kimi Linear matter for hardware: the log-decay is lower-bounded through a scaled sigmoid instead of an unbounded negative softplus, which lets every causal tile in the chunkwise computation run as a dense tensor-core matrix multiply, and the output gate becomes an input-dependent full-rank projection. The MLA layers keep DeepSeek-V2-style compressed KV so global attention stays affordable at 1M tokens.

Attention Residuals. A standard residual stream compresses everything before layer L into one vector, which the report compares to an RNN over depth. AttnRes lets layer L attend over the outputs of all earlier layers with data-dependent weights. Full AttnRes is expensive, so K3 uses a block variant: 93 layers are grouped into blocks of 12, and attention runs over block-level summaries, which the report says recovers most of the benefit.

Stable LatentMoE. LatentMoE routes tokens to experts through a 3,584-dimensional latent, half the 7,168 hidden width, so expanding to 896 experts and 16 active does not scale communication with routing multiplicity. K3 adds an RMSNorm between expert aggregation and the up-projection (Normalized LatentMoE), a bounded SiTU-GLU activation, and Quantile Balancing, a routing rule derived from an assignment relaxation that keeps expert loads balanced at this sparsity.

Per-head Muon. K3 trains its matrices with Muon, but for attention projections it orthogonalizes each head’s block of the momentum separately rather than the whole Q, K or V matrix, on the grounds that whole-matrix orthogonalization couples heads that should update at different scales. The same idea appears independently in Pion for RL fine-tuning.

Key results

  • Scaling efficiency: fitted scaling laws in Figure 7 show K3 reaching a given loss with roughly 2.5x less compute than K2; the report credits KDA, AttnRes, Stable LatentMoE and the data recipe jointly, without a per-component split.
  • Coding, all at max effort (Table 2): Terminal-Bench 2.1 88.3 (GPT-5.6 Sol 88.8, Claude Fable 5 88.0, Opus 4.8 84.6), FrontierSWE 81.2 (Fable 5 86.6, Sol 71.3), DeepSWE 67.5 (Sol 73.0, Fable 5 70.0), ProgramBench 77.8 (best), SWE-Marathon 42.0 (best; Opus 4.8 40.0), SciCode 58.7 (Fable 5 60.2).
  • Agentic: BrowseComp 91.2 (best; Sol 90.4, Fable 5 88.0), MCPMark-Verified 94.5 (best), AutomationBench 30.8 (best), GDPval-AA v2 Elo 1686 (Fable 5 1747, Sol 1736), AA-Briefcase 1548 (Fable 5 1583), Agents’ Last Exam 28.3 (Sol 29.6), JobBench 54.3 (Fable 5 57.4).
  • Reasoning and knowledge: GPQA Diamond 93.5 (Sol 94.1, Fable 5 92.6), HLE-Full 43.5 without tools and 56.0 with (Fable 5 53.3 / 63.0), AA-LCR 74.7 (best).
  • Third-party (as of July 23, 2026): Artificial Analysis Intelligence Index v4.1 57.1, rank 4 of 580; Vals Index 74.7, rank 2 of 39; WebDev Arena Elo 1,678, rank 1 of 99 and the first open model to lead it; Text Arena 1,486, rank 8 of 200; Agent Arena 9.1, rank 4 of 37.
  • Scale of the run: 2.8T parameters, 1M-token agentic RL with persistent rollout and sandbox state, MoonEP expert-parallel training described as perfectly load-balanced, and MXFP4 quantization-aware post-training for deployment.

Limits and open questions

The 2.5x figure is a fitted scaling-law comparison against Moonshot’s own previous model, not a controlled ablation of each architectural change; the report does not separate KDA’s contribution from AttnRes or the MoE changes. Benchmark tables run K3 at max effort and temperature 1 against proprietary models at their maximum settings, and several agentic numbers depend on harnesses (Kimi Code for K3, Claude Code for others on Kimi Code Bench). K3 trails Claude Fable 5 on FrontierSWE by 5.4 points and on GDPval-AA by 61 Elo, so “frontier-level” is accurate for the agentic and browsing suites and softer for the hardest software-engineering ones. Where each architectural piece sits against DeepSeek-V4’s and Qwen3.8-Next’s answers to the same problems is in Kimi K3 Architecture Explained.

This paper is one of the sources behind Muon vs AdamW, Nemotron 3 Architecture Explained, Qwen3.8-Next Architecture Explained.

FAQ

What is the Kimi K3 architecture?

A 93-layer, 2.78T-parameter MoE with 104.2B active parameters. Token mixing alternates three Kimi Delta Attention layers with one gated MLA layer; Attention Residuals let layers read from all earlier layers in blocks of 12; Stable LatentMoE routes each token to 16 of 896 experts plus 2 shared experts through a 3,584-dimensional latent; matrices are trained with per-head Muon.

How does Kimi K3 compare with Kimi K2?

K2 had 1.04T total and 32.6B active parameters, 61 layers, 384 experts with 8 active, MLA-only attention and a 128K training context. K3 has 2.78T total, 104.2B active, 93 layers, 896 experts with 16 active, the KDA and MLA hybrid, and a 1M context, with a reported 2.5x gain in scaling efficiency.

What are Kimi K3’s benchmark results against Claude and GPT?

At max effort K3 scores 88.3 on Terminal-Bench 2.1 (GPT-5.6 Sol 88.8, Claude Fable 5 88.0), 91.2 on BrowseComp (best in the table), 93.5 on GPQA Diamond and 81.2 on FrontierSWE (Fable 5 86.6). It ranked first on WebDev Arena and fourth on the Artificial Analysis Intelligence Index at release.

What is Kimi Delta Attention?

A linear attention layer based on the delta rule with a channel-wise forget gate, a lower-bounded log-decay implemented with a scaled sigmoid so chunkwise computation runs on dense tensor-core tiles, and a full-rank input-dependent output gate. K3 uses three KDA layers for every global gated MLA layer.