Kimi K3 Architecture Explained: KDA, Attention Residuals and LatentMoE

Architecture at a glance

The Kimi K3 report organizes the model around three axes of mixing: across tokens, across layers, and across channels. Along tokens, each block stacks three Kimi Delta Attention (KDA) layers and one gated Multi-head Latent Attention layer, with one more gated MLA layer at the very end so the final layer is always global attention. Along layers, Attention Residuals replace the single accumulated residual with attention over the outputs of earlier layers. Along channels, Stable LatentMoE routes each token to 16 of 896 experts plus 2 shared experts through a 3,584-dimensional latent, half the 7,168 hidden width. A from-scratch vision tower, MoonViT-V2, feeds the same backbone. Matrices are trained with Muon, with attention projections orthogonalized per head. The totals: 93 layers, 2.78T parameters, 104.2B active, 1M-token training context.

Key numbers

Dimension	Kimi K2	Kimi K3	Source	Same harness?
Total / active parameters	1.04T / 32.6B	2.78T / 104.2B	K3 report, Table 1	n/a
Layers / attention heads	61 / 64	93 / 96	Table 1	n/a
Attention	MLA in every layer	3 KDA : 1 gated MLA, plus a final gated MLA	Table 1, Section 2.1	n/a
Routed experts / active per token / shared	384 / 8 / 1	896 / 16 / 2	Table 1	n/a
Latent MoE dimension	none	3,584 (0.5x hidden)	Table 1	n/a
MoE hidden per expert	2,048	3,072	Table 1	n/a
Training context	128K	1M	Table 1	n/a
Depth mixing	standard residual	block Attention Residuals, 8 blocks of 12 layers	Section 2.2	n/a
Optimizer	Muon	per-head Muon on attention projections	Section 2.5	n/a
Scaling efficiency	1x	about 2.5x (fitted scaling laws)	Figure 7	Yes, same lab’s fits
WebDev Arena / AA Intelligence Index rank at release	n/a	1 of 99 (Elo 1,678) / 4 of 580 (57.1)	Third-party, July 23 2026	n/a

The 2.5x is a fitted-curve comparison between two generations of the same lab’s models with data and recipe changes included; the report does not ablate KDA, AttnRes and the MoE changes separately.

Token mixing: what KDA is and why 3:1

KDA is a linear attention layer that keeps a fixed-size recurrent state and updates it with the delta rule, the error-correcting write that overwrites what the state currently predicts for a key rather than accumulating on top of it, gated per channel so different dimensions forget at different rates. It descends from the same family as Mamba’s selective state space and Gated DeltaNet, and it inherits their property: constant per-token decode cost and no growing KV cache. K3 changes two things relative to Kimi Linear. The log-decay is bounded from below by a scaled sigmoid instead of an unbounded negative softplus, which lets every causal tile of the chunkwise computation run as a dense tensor-core matrix multiply rather than an explicit per-position loop. And the output gate becomes a full-rank input-dependent projection after head-wise RMSNorm.

The 3:1 ratio exists because a fixed-size state cannot do exact retrieval. Global attention layers with compressed KV keep every token addressable; three cheap layers between each of them hold the long-range cost down. The final global layer guarantees the last computation before the head can look at everything. Qwen3.8-Next reaches the identical 3:1 split with Gated DeltaNet and reports the controlled ablation behind it: a GDN hybrid averaged 53.81 against 49.87 for full attention and 51.15 for a sliding-window hybrid at 25B-A3B. K3 does not publish that ablation, so Qwen’s number is the best available evidence that the ratio is right.

Depth mixing: Attention Residuals

A residual stream compresses everything before layer L into one vector, which the report compares to an RNN over depth. Attention Residuals let layer L attend, with data-dependent weights, over the outputs of all earlier layers, the way self-attention replaced recurrence over positions. Full AttnRes over 93 layers is expensive, so K3 partitions the layers into 8 blocks of 12 and attends over block-level representations, plus the embedding layer; the report says this recovers most of the benefit and describes an online-softmax implementation to keep inference cost down. It is a different answer from DeepSeek V4’s manifold-constrained hyper-connections and Qwen3.8-Next’s gated four-branch residual: all three widen or re-wire the residual path, but K3 is the only one that makes it attention.

Channel mixing: Stable LatentMoE

Increasing both the expert pool and the number of active experts widens the space of specializations, but in a conventional MoE each selected expert receives the full hidden width, so communication grows with routing multiplicity. LatentMoE separates the model width from the routed-expert width: shared experts keep a full-width path, routed experts operate on the 3,584-dimensional latent. K3 adds three stabilizers for this sparsity. An RMSNorm between expert aggregation and the up-projection keeps the routed branch’s scale from varying with which experts fired. SiTU-GLU, a sigmoid-tanh gated linear unit, bounds both branches of the activation. Quantile Balancing replaces auxiliary-loss balancing with a routing rule derived from an assignment relaxation and estimated with histograms, keeping loads even at 16 of 896.

Optimizer: per-head Muon

K3 trains matrices with Muon as K2 did, but for Q, K and V projections it partitions the momentum along the head dimension and orthogonalizes each head’s block separately. The argument is that whole-matrix orthogonalization couples heads whose norms and roles differ; the Pion paper reaches the same per-head design independently for RL fine-tuning. DeepSeek V4 runs whole-matrix Muon with a logit-explosion guard, and Qwen3.8-Next keeps the MoE router on AdamW after finding Muon destabilized it.

Limits and open questions

The architecture is presented as a package with a fitted 2.5x efficiency gain; no per-component ablation is published, so the size of each contribution is unknown. KDA’s 3:1 ratio borrows its evidence from Qwen’s GDN ablation at a smaller scale. Block AttnRes is described as recovering most of full AttnRes without the numbers. And the 1M-context and agentic results, strong as they are, run at max reasoning effort under Moonshot’s harness, so they characterize the model plus its post-training rather than the architecture alone.

FAQ

What is the Kimi K3 architecture in one paragraph?

A 93-layer, 2.78T-parameter MoE with 104.2B active parameters. Each block has three Kimi Delta Attention layers and one gated MLA layer; Attention Residuals let layers read all earlier layers in blocks of 12; Stable LatentMoE routes tokens to 16 of 896 experts through a 3,584-dimensional latent; matrices are trained with per-head Muon; a native MoonViT-V2 tower supplies vision.

How does Kimi K3’s attention differ from DeepSeek V4’s?

K3 uses linear delta-rule attention (KDA) for three of every four layers and compressed global MLA for the fourth, so most layers have no growing KV cache. DeepSeek V4 keeps attention in every layer but compresses groups of tokens into single KV entries and attends sparsely to the top-k, shrinking the cache to 10% of V3.2’s at 1M tokens. Both add a sliding window for local detail.

What are Attention Residuals in Kimi K3?

A depth-mixing mechanism where each layer attends over the outputs of preceding layers with data-dependent weights instead of reading one accumulated residual vector. K3 uses a block variant over 8 blocks of 12 layers to keep cost down.

Why does Kimi K3 use 896 experts with only 16 active?

Because LatentMoE routes tokens through a half-width latent, so expanding the expert pool does not scale communication with the number of active experts, and Normalized LatentMoE, SiTU-GLU and Quantile Balancing keep training stable at that sparsity.