Research Papers Compare
Architecture and method explainers grounded in the source papers, with the key numbers.
Language Models · DeepSeek V4
DeepSeek V4 keeps V3's MoE and multi-token prediction, rebuilds attention as a compress-then-sparsify hybrid, widens the residual stream under a doubly stochastic constraint, and trains most matrices with Muon.
Language Models · Kimi K3
Kimi K3 uses three delta-rule linear-attention layers per global MLA layer, attention over earlier layers for depth, and 16 of 896 experts through a half-width latent, for 2.5x K2's scaling efficiency.
Long Context · KV cache compression
The KV cache, not the weights, runs out of memory at long context. Four 2026 routes attack it: 2-bit quantization for long decodes, lookahead offloading, 1:16 soft-token compression and fixed-size memory.
Language Models · Nemotron 3
Nemotron 3 Ultra keeps most token mixing in Mamba-2 layers, uses attention with only 2 KV heads, routes tokens to 22 of 512 experts, and trained in NVFP4; that is why it decodes up to 5.9x faster than GLM-5.1.
Language Models · Qwen3.8-Next
Qwen3.8-Flash-Next activates 6B of 125B parameters plus 51B host-memory n-gram tables; a 3:1 GDN hybrid, sparse attention, a gated four-branch residual and Muon give Qwen3.7-Plus quality at a ninth of the FLOPs.
Efficient AI · Speculative decoding
A small drafter proposes several tokens, the big model verifies them in one pass, and rejection sampling keeps the output exact. The whole game is how many draft tokens survive, and 2026's gains come from the drafter.