MTP vs NITP: Two Ways to Escape One-Token-At-A-Time Supervision

The shared complaint

Standard next-token prediction (NTP) supervises a language model with one discrete label per position: the ID of the next token. Two research lines agree this is wasteful, and each attacks a different half of the waste.

Multi-token prediction (MTP), the approach behind Bebop, complains that NTP uses its compute serially. At every position the model emits exactly one token’s distribution and stops. An MTP head instead predicts the next few tokens (positions t+1 through t+γ) from the same hidden state. At inference those extra heads draft tokens in bulk and the main model verifies them, skipping forward passes for every accepted draft. The payoff is wall-clock, and in RL training, where rollout generation dominates the step time, that payoff is large.

Next implicit token prediction (NITP), from the NITP paper, complains that NTP’s target is too sparse. A single token ID pins down almost nothing of the rich hidden state at each position, so most of the representation’s geometry is shaped only indirectly and can collapse toward low-rank structure. NITP adds a second, continuous objective: at position t, predict the shallow-layer hidden representation of position t+1. The payoff is representation quality, bought during training and dropped before inference.

Both are auxiliary objectives bolted onto NTP. Neither replaces the cross-entropy readout. Beyond that they diverge in nearly every detail, which is the point of this comparison.

Key numbers

Measurement	MTP (Bebop)	NITP	Setting and source
What is predicted	Discrete token IDs at t+1 … t+γ	Continuous shallow-layer embedding of t+1 (~20% depth, stop-gradient)	Bebop Sec. 3 / NITP Sec. 3
Loss that carries the result	e2e total-variation loss	Cosine similarity (MSE diverges)	Bebop Fig. 8 / NITP ablation
Headline accuracy effect	None; output is provably unbiased under rejection sampling	MMLU-Pro 15.29 → 21.00 (+5.71), 13-task avg +2.67	NITP, 9B MoE (1B activated)
Headline throughput effect	Acceptance held near 95% (~+10 pts over CE); RL latency 1.5–1.8x faster, agentic rollouts 2.4x	None measured; training-time objective only	Bebop Figs. 8–9
Entropy robustness	Acceptance-vs-entropy slope −1.68 (CE) → −0.06 (TV)	Not tested in an RL entropy regime	Bebop Fig. 8
Cost at inference	Zero; the MTP head is the drafter	Zero; the implicit-token head is dropped after training	Both papers
Cost during training	One-time pre-RL TV training pass, needs fused full-vocab kernel	~2% extra training FLOPs, negligible wall-clock	Bebop Sec. 5 / NITP Sec. 4
Where the gain decays	OOD tasks: +8.0 SWE-Bench in-distribution but +2.3 MT-Bench OOD	Above 45B MoE untested; gains largest on knowledge and reading tasks	Both papers’ limits
Scale tested	Qwen3.5 / Qwen3.6 / Qwen3.7 RL runs	1.9B–45B MoE, 0.5B–3B dense	Both papers

Read the table for what it does not contain: a single run that trains both objectives together and measures both accuracy and acceptance. That experiment does not exist in either paper. MTP and NITP have never been compared under one harness, so every row above comes from a different model family and a different measurement.

How each one works

MTP: draft γ tokens, verify, accept

Bebop’s MTP head drafts γ tokens ahead during rollout. The novel problem it solves is specific to RL: the paper proves that the draft acceptance rate under greedy checking falls linearly as policy entropy rises, following α ≈ a − b·H(p). RL training deliberately keeps entropy high for exploration, so the dynamics that make RL work directly erode the speedup, exactly when rollouts are the bottleneck.

Bebop’s fix has two separable parts. Rejection sampling replaces greedy checking: accept draft token v with probability min(p(v), q(v)), which makes the accepted output exactly match the target distribution no matter how bad the draft is, and makes acceptance equal 1 − d_TV(p, q). That alone does not break the entropy bound: a cross-entropy-trained head still degrades with the same −1.68 slope. The load-bearing piece is a total-variation training loss that minimizes d_TV directly, whose gradient is bounded and proportional to q_j, so optimization effort lands where probability mass actually is. That flattens the entropy slope to −0.06.

The loss ablation quantifies how sharp the choice is. Under identical rejection sampling: KL over CE adds +0.0 to +0.2 acceptance points, reverse KL adds +1.0 to +1.3, per-step TV adds +2.4 to +5.2, and the end-to-end TV loss adds +3.0 to +8.0. The closer the loss gets to directly optimizing the acceptance quantity, the better it performs.

NITP: supervise the representation, not just the readout

NITP keeps the NTP head untouched and trains a parallel head to predict the shallow-layer hidden state one position ahead. Three design choices matter, and each was selected by ablation rather than intuition.

First, the target comes from a shallow layer at roughly 20 percent of model depth, passed through a stop-gradient so it acts as a stable teacher rather than a moving target; shallow targets clearly beat middle- and deep-layer ones. Second, the prediction is shifted: the model at position t matches the implicit token at t+1, mirroring NTP’s look-ahead; aligning to the current position substantially underperforms. Third, the loss is cosine similarity, and the paper reports MSE caused divergence, a sign the method is sensitive to target normalization and should be treated as a tuning risk, not a free lunch.

The reported mechanism is geometric: NITP-trained representations show higher effective rank, supporting the claim that the dense target counters representation collapse that pure NTP leaves under-constrained.

What you are actually buying in each case

MTP buys throughput and nothing else. Bebop is explicit that accuracy does not change, because rejection sampling guarantees the sampled output equals the target distribution. If you saw “breaking entropy bounds” and read it as a reasoning result, it is not; the entropy in question is the policy’s sampling entropy, and the bound is on draft acceptance. The business case is pure: agentic RL rollouts run 2.4x faster, general RL steps 1.5–1.8x faster, and the only training cost is a one-time TV pass on the MTP head before RL starts. Notably, continuing to train the head with CE during RL erodes the gain back toward baseline, while continuing with TV adds nothing, so pre-RL training is sufficient, which also removes the memory cost of online head training.

NITP buys accuracy and nothing else. There is no inference-time component: the implicit-token head is dropped after training, deployed models run identically to an NTP baseline, and the paper measures no latency effect. The business case is a 2 percent training-FLOPs surcharge for a +2.67-point average across 13 benchmarks on a 9B MoE, with the largest single gain at +5.71 on MMLU-Pro. A 3B dense model also gained +1.35 on average, so the objective is not MoE-specific.

Limits and open questions

MTP’s acceptance gain is in-distribution. The same head that lifts SWE-Bench acceptance from 75.1% to 83.1% gains only +2.3 on out-of-distribution MT-Bench, so a TV-trained head may not transfer to traffic far from the RL data. Acceptance still drops with generation position and with sampling temperature since both raise entropy, and long agentic rollouts show periodic acceptance fluctuation, the “bubble problem.” The full-vocabulary TV loss also needs fused kernels; a top-K = 20,000 approximation slows convergence and hurts performance. Everything is validated on Qwen models with GRPO, so transfer to other families and RL algorithms is assumed, not shown.

NITP’s largest reported model is 45B MoE with 5.5B activated, well below frontier scale, so whether the 2-percent-for-multi-points trade survives at hundreds of billions of parameters is unverified. The benchmark gains concentrate on knowledge and reading-comprehension tasks; transfer to generation quality, long context, or reasoning chains is not measured. And the MSE-divergence result means the cosine-loss choice is not cosmetic: teams reimplementing it should expect to spend tuning effort on target normalization.

When to use which

The choice is a bottleneck question, not a quality question. If your cost center is generating tokens (RL rollouts, high-volume inference) and you can add a draft head, MTP with a TV loss is a pure speedup with no accuracy give-up; the wider drafting mechanics around it are covered in Domino speculative decoding and on-policy draft training in Draft-OPD. If you are pretraining and want the cheapest measured accuracy-per-FLOP in this pair of papers, NITP’s ~2 percent surcharge for multi-point benchmark gains is the better trade.

Can you use both? Nothing in either paper rules it out, and the mechanisms are orthogonal: one reshapes the MTP draft head’s loss, the other adds a representation-level target to the main model. But orthogonality in design is not evidence of a combined effect, and no published result trains NITP plus a TV-trained MTP head and reports both benchmark accuracy and acceptance. Treat “combine them” as a hypothesis, not a recommendation.

FAQ

Is MTP the same thing as NITP?

No. MTP predicts several discrete token IDs ahead and uses those predictions to draft and verify tokens at inference, chasing throughput. NITP predicts one continuous representation ahead as an extra training target, chasing representation quality. They share only the premise that NTP’s supervision is too thin.

Does multi-token prediction make the model more accurate?

In Bebop’s setup, no. Rejection sampling accepts a draft token with probability min(p(v), q(v)), which makes the sampled output exactly match the target model’s distribution regardless of draft quality. The gains are in rollout speed: up to 1.8x faster asynchronous RL, 2.4x on agentic rollouts, with benchmark scores unchanged.

How much does NITP cost?

About 2 percent additional training FLOPs with negligible wall-clock impact, and zero inference cost, because the implicit-token head is dropped after training. The headline return is +5.71 points on MMLU-Pro and a +2.67-point average across 13 benchmarks on a 9B MoE.

Why does Bebop’s TV loss beat cross-entropy for MTP heads?

Because acceptance under rejection sampling equals 1 − d_TV(p, q), and the TV loss directly minimizes that quantity with a gradient proportional to the draft probability q_j, so effort goes where probability mass is. CE and KL produce a uniform per-token mismatch that accumulates across the exponentially larger effective support a high-entropy policy opens up, so acceptance degrades with a −1.68 entropy slope either way; TV flattens it to −0.06.

Which one should I use?

If your bottleneck is generating tokens (RL rollouts, high-volume inference) and you can add a draft head, MTP with a TV loss is a pure speedup with no accuracy give-up. If you are pretraining and want the cheapest measured accuracy-per-FLOP in this pair of papers, NITP’s ~2 percent surcharge for multi-point benchmark gains is the better trade. Different bottlenecks, different tools.