Efficient AI · Language Models · Transformers
Speculative Decoding Explained: Draft, Verify and the Acceptance Ceiling
A small drafter proposes several tokens, the big model verifies them in one pass, and rejection sampling keeps the output exact. The whole game is how many draft tokens survive, and 2026's gains come from the drafter.
How it works
Autoregressive decoding spends one full forward pass of the target model per token, and at small batch sizes that pass is dominated by reading the weights, not by arithmetic. Speculative decoding fills that idle compute. A cheap drafter proposes a block of k candidate tokens. The target model then scores all k positions in a single forward pass, exactly as it would score a prompt, and accepts the longest prefix it agrees with.
Agreement is defined so the output distribution never changes. With greedy verification a token is accepted only if it equals the target’s argmax. With rejection sampling, the standard lossless rule, a draft token v drawn from the draft distribution q is accepted with probability min(1, p(v)/q(v)) under the target distribution p, and on rejection a corrected token is sampled from the residual. The expected acceptance probability per position becomes 1 minus the total variation distance between p and q. Accepted tokens are provably distributed as if the target had generated them alone; the only thing that changes is wall-clock time.
The cost model follows. If the drafter is nearly free and the target accepts tau tokens per verification step on average, the speedup approaches tau. Every method in this explainer is an attempt to raise tau, lower the drafter’s cost, or keep tau from decaying.
Key numbers
| Measurement | Value | Setting and source | Same harness? |
|---|---|---|---|
| Qwen3-4B thinking, speedup / acceptance length | Draft-OPD 4.86x, tau 5.96; DFlash 4.33x, 5.51; EAGLE-3 3.87x, 5.33 | temperature 0, Draft-OPD paper | Yes |
| Qwen3-8B thinking | Draft-OPD 4.89x, tau 5.73; DFlash 4.34x, 5.19; EAGLE-3 4.06x, 5.64 | Same | Yes |
| Aggregate gain from on-policy drafter training | +23% over EAGLE-3, +13% over DFlash | matched compute, Draft-OPD | Yes |
| End-to-end speedup, decoupled parallel drafter | up to 5.49x with a Transformers backend; up to 5.8x throughput under SGLang | Domino paper | Own baselines |
| Acceptance under RL with a TV-trained MTP head | about 95%, roughly +10 points absolute over cross-entropy training | Qwen3.5-35A3B, gamma = 3, Bebop | Yes |
| Acceptance vs policy entropy slope during RL | -1.68 (greedy, or rejection sampling with CE) → -0.06 (rejection sampling with TV loss) | Bebop, Figure 8 | Yes |
| Per-domain acceptance, CE → e2e TV | SWE-Bench 75.1 → 83.1; agentic 90.3 → 97.0; math 75.0 → 78.0; MT-Bench 65.3 → 67.6 | Bebop | Yes |
| RL rollout latency with MTP | 1.5 to 1.8x lower per step; 2.4x on agentic rollouts; up to 1.8x end-to-end async RL | Qwen3.5 / 3.6 / 3.7, Bebop | Yes |
| Speculative decoding inside a diffusion LM | +5.04 points average accuracy, 3.86x speedup | FeF-DLLM on GSM8K, MATH, HumanEval, MBPP | Within diffusion LMs |
| Output distribution | unchanged, by construction of rejection sampling | all methods | n/a |
Speedup figures are sensitive to batch size, hardware and the base model; the two Qwen3 rows are the cleanest like-for-like numbers because three drafters were run on identical targets.
Where the ceiling comes from
Acceptance length is bounded by how well the drafter tracks the target, and three things erode it.
Offline training. Drafters such as EAGLE-3 and DFlash are trained by supervised fine-tuning on the target’s transcripts, so they never see the states their own drafting creates. Draft-OPD shows the symptom, acceptance length plateauing during SFT, and fixes it with target-assisted rollouts plus replay from the positions where drafts were rejected, using forward KL on accepted tokens and reverse KL on rejected ones. That is where the 23% over EAGLE-3 comes from.
Parallel drafting. Drafting all k tokens in one shot is fast but each position is predicted blind to its neighbors, so the block is internally inconsistent and the target rejects more of it. Autoregressive drafters fix consistency at the cost of k sequential passes. Domino splits the two jobs: a parallel backbone proposes the block, a small causal head rewrites it using the confirmed prefix, and a base-anchored curriculum keeps the backbone from degrading while the head learns. The reported 5.49x and 5.8x come from a higher accepted-tokens-per-step ratio, not from a cheaper draft.
Entropy. During reinforcement learning the policy’s entropy rises on purpose, and Bebop proves the acceptance rate under greedy checking is linearly bounded by that entropy: alpha is roughly a minus b times H(p). Switching to rejection sampling makes the decay smooth but does not break the bound, and neither does training the draft head with KL or cross-entropy. Training it with a total-variation loss, the exact quantity rejection sampling depends on, flattens the slope from -1.68 to -0.06 and holds acceptance near 95% through RL.
What this means for diffusion decoders
The same verify-then-accept idea transfers outside autoregression. Diffusion language models unmask many tokens per step but predict them independently, which introduces factorization error. FeF-DLLM makes the clean-token posterior exactly prefix-conditioned, which would be sequential, then recovers parallelism with speculative decoding inside each denoising step: 5.04 points more accuracy on average and a 3.86x speedup over the sequential exact version.
Limits and open questions
Every speedup above is measured against the authors’ own baselines on one model family, mostly Qwen3; cross-family transfer is assumed. Draft-OPD trains on 4,096-token sequences and evaluates at 8,192. Domino’s abstract does not report acceptance length head-to-head against EAGLE or Medusa on a fixed setup. Bebop’s TV loss needs a fused full-vocabulary kernel, and a top-20,000 approximation slows convergence; its gains are in-distribution (+8.0 on SWE-Bench acceptance, +2.3 on MT-Bench). And none of the papers benchmark against non-MTP drafters under the same RL entropy conditions, so “TV is the best loss for an MTP head” is established, “MTP is the best drafter” is not.
FAQ
Is speculative decoding lossless?
Yes when verification uses rejection sampling: a draft token is accepted with probability min(1, p/q) and a corrected token is sampled on rejection, so accepted output is distributed exactly as the target model alone would produce. Greedy verification is lossless against greedy decoding of the target. Only speed changes.
How much speedup does speculative decoding give in 2026?
On Qwen3 thinking models at temperature 0, Draft-OPD reaches 4.86x on 4B and 4.89x on 8B with acceptance lengths near 6, against 3.87x and 4.06x for EAGLE-3. Domino reports up to 5.49x with a Transformers backend and 5.8x throughput under SGLang. Numbers vary with batch size and hardware.
Why does speculative decoding acceptance drop during RL training?
Because RL raises policy entropy for exploration, and acceptance under greedy or CE-trained drafting falls linearly with entropy, slope about -1.68 in Bebop’s measurements. Training the draft head with a total-variation loss under rejection sampling flattens the slope to -0.06 and keeps acceptance near 95%.
What is the difference between EAGLE-style and Medusa-style drafters in speculative decoding?
EAGLE-style drafters are autoregressive, so each draft token sees the ones before it, giving high acceptance at the cost of one sequential pass per token. Medusa-style heads draft the whole block in parallel, cheaply but inconsistently. Domino combines them: a parallel backbone proposes, a small causal head repairs the block.