Efficient AI · Transformers · Language Models
Test-Time Training vs Sleep: Learning After the Training Run
SD3.5-T5 uses a TTT layer's per-token gradient step to convert Softmax attention for about 1 hour, hitting 1.32x-1.47x speedup; Sleep shows test-time training fails at 10% ARC and moves learning offline to 80%.
The name collision worth untangling
Search “test-time training” in 2026 and you hit two papers that barely overlap. One is an architecture paper: Linearizing ViT with Test-Time Training, where a Test-Time Training (TTT) layer takes a small gradient step inside every forward pass so that a pretrained Softmax transformer can be converted into a linear-attention model. The other is a memory paper: Language Models Need Sleep, where “test-time training” appears only as the baseline that fails, at 10% on ARC few-shot, and the actual proposal is to stop learning at inference entirely and do the learning offline, between sessions, while the model is “asleep.”
Both are about the same itch: the training run ends, deployment begins, and the model still needs to adapt. They just disagree about when and where that adaptation should happen. Reading them together answers the real question, which is not “what is TTT” but “which kind of post-training learning do I actually want.”
Side one: the TTT layer as an architecture (SD3.5-T5)
The SD3.5-T5 paper’s problem is cost, not memory. Linear-complexity attention removes the quadratic term that makes high-resolution diffusion slow, but training a linear model from scratch costs as much as the original model. The shortcut everyone wants is to inherit a pretrained transformer’s weights and fine-tune briefly, and that shortcut kept failing: in the paper’s ImageNet probe, plain linear attention fine-tuned from a DeiT backbone reaches 63.30% Top-1, LiT-style grafting 69.52%, CLEAR 68.66%, and Hedgehog diverges to NaN, all against 72.05% for the Softmax DeiT-Tiny.
The fix is a structural match rather than a clever kernel. Softmax attention can be rewritten as a two-layer transform, query through a key-built matrix, a nonlinearity, then a value-built matrix. A TTT layer has exactly this shape: its inner model is a small two-layer MLP whose first layer is adjusted by one gradient step at inference, which reads as query, nonlinear middle, output. Because the two architectures line up term for term, the pretrained Q, K, V, and O projections plug straight in. The ablation makes the point numerically: on a DiT-S/2 generator, full weight inheritance gives FID 68.52, MLP-only inheritance 89.71, attention-only 93.18. The attention weights are where the transferable knowledge lives.
Two patches make the inheritance trainable. Pretrained keys carry a large constant shift (measured shift ratio about 0.5 across 5K ViT activations, versus 0.07 for random init) that Softmax cancels but TTT does not, so the inner optimization explodes without key instance normalization; that single change is the difference between NaN and 71.19% Top-1 on DeiT-Tiny. A small locality module, a depthwise convolution on queries and keys plus a neighborhood-attention term, lifts it further to 72.06%, passing the 72.05% Softmax baseline. Its full result table is in Key numbers below.
Side two: Sleep, where test-time training loses (Google Research)
The Sleep paper’s problem is memory, not speed. A model that handles a long conversation holds new facts only in its activations and context window; nothing is written back to the weights, so the next session starts from scratch. The authors frame in-context learning as “selective and retrieval-dependent,” then propose a two-stage offline recipe. Consolidation distills recent context into the network via upward distillation, a smaller model that absorbed the context teaches a larger one, combining generalized knowledge distillation with an RL-based learning-to-imitate objective, with the parameter count expanded to make room. Dreaming then generates synthetic practice data, scores candidate dreams by gradient-based importance, and self-improves on them with no human labels. The two phases are deliberately separate because iteratively self-modifying in one loop “might cause catastrophic forgetting.”
Key numbers
SD3.5-T5 (TTT layer conversion)
| Result | SD3.5-T5 (linearized) | Baseline it compares against |
|---|---|---|
| GenEval overall (SD3.5) | 0.69 | 0.70 fine-tuned Softmax, 0.66 original SD3.5 |
| DPG-Bench (SD3.5) | 84.43 | 83.83 original SD3.5-Medium |
| Inference latency, 1024x1024, 1x H20 | 19s (1.32x) | 25s |
| Inference latency, 2048x2048, 1x H20 | 1.47x gain | quadratic term dominates at 2K |
| Conversion cost | 3000 steps, batch 64, about 1 hour on 4 H20 | 1400 epochs from scratch for DiT |
| DiT5-XL/2 FID after conversion | 2.48 | 2.27 original, at 0.57% of training cost |
| DeiT-Tiny Top-1 after conversion | 71.19% (72.06% with locality) | 72.05% Softmax; 63.30% plain linear |
The honest reading: quality is parity within noise, not a win, and the speedup is resolution-gated. Below 1K the linear layer has little quadratic cost to remove and can even lose. The claim that survives is conversion economics: about one hour of fine-tuning buys a real high-resolution speedup without throwing away the pretrained model.
Sleep (offline consolidate-and-dream)
| Benchmark | Sleep | The baselines it beats |
|---|---|---|
| ARC few-shot | 80% success | 72.5% SEAL, 10% test-time training, 0% in-context learning |
| AIME-24 (Qwen3-8B) | 79.2% | +2.6% over OPSD baseline |
| AIME-25 / HMMT-25 (Qwen3-8B) | 69.0% / 46.1% | 53.2% / 40.2% / 29.3% for the 1.7B model |
| SQuAD knowledge incorporation | 48.9% one passage, 46.2% at n=200 | 46.7% / 43.2% for SEAL |
| BABILong long-context retrieval | near-perfect at 10M tokens | Titans and ARMT degrade sharply beyond 1M |
Two readings matter here. First, the paper’s own table is the clearest published verdict on naive test-time training: on the exact capability it targets, persistent few-shot learning, per-token TTT scores 10% against sleep’s 80%. Second, the strongest results ride on a custom multi-frequency memory architecture called Hope, where different parameter groups update at different rates across sleep cycles, so it is not yet clean how much comes from the sleep schedule versus the architecture underneath. The +2.6% AIME-24 margin over OPSD is real but narrow, and the paper reports no compute cost for a sleep cycle, which leaves the practicality of running this between deployments unestablished.
The cross-paper comparison
| Dimension | TTT layer (SD3.5-T5) | Sleep (consolidate-and-dream) |
|---|---|---|
| What “test time” means here | Every forward pass: one gradient step inside the layer | Nothing happens at inference; learning is offline between sessions |
| What gets updated | The TTT layer’s inner model, per token | The full network’s weights, per sleep cycle |
| What the model keeps afterwards | Nothing persistent; adaptation resets each sequence | Durable parameter memory across sessions |
| One-time setup cost | 3000 steps, about 1 hour on 4 H20 | Unreported; consolidation plus dreaming cycles |
| Recurring inference cost | Lower than Softmax at 1K-2K (1.32x-1.47x) | Same as base model at inference |
| Headline quality delta | GenEval 0.69 vs 0.70 fine-tuned Softmax; DeiT 72.06% vs 72.05% | ARC 80% vs 72.5% SEAL and 10% TTT; AIME-24 79.2% |
| Main failure mode | NaN from uncentered keys; parity-only quality | Catastrophic forgetting if phases are fused; unreported sleep cost |
| The old approach it replaces | Retraining linear attention from scratch | In-context learning and SEAL-style self-editing |
The table makes the disagreement concrete. SD3.5-T5 keeps learning at inference but forgets everything between sequences; Sleep refuses to learn at inference and is designed around keeping things between sessions. “Test-time training” as a blanket term hides that these are almost complementary designs, and a system could in principle use a TTT layer inside a model that sleeps between deployments, since the layer’s per-token adaptation is exactly the kind of cheap, non-durable learning the sleep framework treats as the “wake” phase.
When to use which
Match the mechanism to the bottleneck. If your cost is inference FLOPs on a model you already own, specifically a Softmax transformer doing high-resolution work, the TTT layer conversion is the rare efficiency recipe with a worked inheritance story and a measured price: 3000 steps, GenEval parity within noise, 1.32x to 1.47x at 1K and 2K. If your cost is a model that forgets everything the moment the context window closes, the published evidence says per-token test-time training is not the answer, 10% on ARC few-shot against 0% for plain in-context learning is barely a win, and an offline consolidate-and-dream loop is where the durable gains are. And if you are benchmarking “learning at inference” claims, insist on which of the two meanings is being measured, because the SD3.5-T5 numbers and the Sleep numbers are not comparable to anything in between.
Limits and open questions
Neither paper closes its own biggest hole. SD3.5-T5 reports no direct FID for SD3.5, relies on GenEval and DPG-Bench alone, and fine-tunes on Flux-generated synthetic images, so it is open how much of the parity reflects distilling Flux rather than preserving SD3.5; the recipe also needs a 20x learning-rate multiplier on TTT parameters and may not transfer to other backbones without retuning. Sleep reports no compute or wall-clock cost for a sleep cycle, the +2.6% AIME-24 margin is narrow, and whether dreaming on self-generated data drifts over many cycles is explicitly the open question that matters most. And both papers test only their own regime: no linearized text or video transformer, no sleep cycle on an architecture someone else designed.
FAQ
What is test-time training, exactly?
It depends on which 2026 paper you read. In the architecture literature, a TTT layer runs a small gradient step on an inner model inside every forward pass, which gives attention-like behavior with linear complexity; SD3.5-T5 uses it to convert pretrained Softmax transformers. In the continual-learning literature, “test-time training” means adapting model weights while serving; Language Models Need Sleep measures this baseline at 10% on ARC few-shot and argues the learning should move offline instead.
Does test-time training update model weights at inference?
Only the TTT layer’s tiny inner model, and only within a single sequence; those updates are discarded after each forward pass. Sleep takes the opposite position: nothing updates at inference, and all learning happens in an offline consolidate-and-dream phase that writes durable changes into the weights.
Why did earlier attempts to convert Softmax attention to linear attention fail?
Because a fixed linear kernel cannot represent what Softmax learned. LiT-style grafting keeps only the MLP weights (69.52% Top-1 on DeiT-Tiny vs 72.05% Softmax), plain linear attention reaches 63.30%, and Hedgehog diverges to NaN. The TTT layer works because its two-layer inner model is structurally the same shape as Softmax attention, so the pretrained Q/K/V/O projections transfer with their meaning intact, hitting 71.19% and 72.06% with a locality module.
How cheap is the SD3.5-T5 conversion?
3000 fine-tuning steps at batch size 64 on Flux-generated 1024x1024 images, about 1 hour on 4 H20 GPUs. For class-conditional DiT the same recipe costs 8 epochs against a 1400-epoch from-scratch baseline, roughly 0.57% of the training cost, landing FID 2.48 versus the original 2.27.
Does Sleep beat SEAL and test-time training?
On every benchmark the paper reports, yes: 80% versus 72.5% for SEAL and 10% for test-time training on ARC few-shot, and 48.9% versus 46.7% on single-passage SQuAD. The caveat is that the largest long-context wins come from the custom Hope memory architecture running underneath the sleep schedule, and the math margin over OPSD is +2.6% on AIME-24, narrow but consistent.
Can the two approaches be combined?
Nothing in either paper rules it out, and the designs are nearly orthogonal: the TTT layer’s per-token adaptation is cheap and non-durable, which is precisely what the sleep framework expects from its “wake” phase. Whether a TTT-layer model consolidates its inference-time adaptation during sleep is an open question neither paper answers.