Video Generation · Efficient AI · Diffusion Models
Stream-T1 vs VLM-Teacher: +0.28 motion quality or +21.8 RULER points
Stream-T1 searches a frozen streaming model's chunks at inference, lifting 5s VideoAlign motion quality from 0.350 to 0.629; the VLM-teacher loop optimizes a per-instance LoRA and lifts RULER-Bench from 46.4 to 68.2.
The shared bet: buy quality back at inference
Both papers in this comparison start from the same refusal. Retraining a video model is expensive, slow, and often impossible if you only have a released checkpoint. So they ask what a frozen video model can still buy back if you spend compute at inference time instead of gradient steps at training time. That is the whole family called test-time compute, and the two papers spend it in opposite currencies.
Stream-T1 spends search. It wraps a frozen autoregressive streaming video generator, expands several candidate continuations per chunk, scores them with reward models, and prunes. Nothing is ever optimized by backpropagation; the model is a fixed artifact that gets queried more cleverly. The VLM-teacher loop (the paper’s own page) spends gradient steps. It freezes the video model too, but attaches a tiny per-instance LoRA and tunes those few adapter parameters at inference time, graded by a vision-language model that watches the intermediate frames and writes its own reward questions.
The two also answer different complaints. Stream-T1 fixes fidelity drift: the subject morphing and background wandering that accumulate when a streaming model generates chunk by chunk. The VLM-teacher loop fixes reasoning failure: a video model that generates fluent motion but does not actually satisfy the task, like “move each colored object to the star marker of the same color.” Picking between them is less about which number is bigger and more about which failure you have.
How each one works
Stream-T1: beam search wearing video-specific machinery. Three training-free mechanisms cooperate per chunk. Stream-scaled noise propagation initializes each new chunk’s latent by spherically interpolating the previous chunk’s noise (an interpolation factor between -1 and 1) instead of sampling fresh noise, so motion carries forward instead of restarting. Stream-scaled reward pruning is the core: expand several continuations, score them with a short-score from image rewards for per-frame quality and a long-score from video rewards over a sliding window of 10 chunks for temporal coherence, and prune with a dynamic threshold that shifts weight from frame quality toward consistency as the clip grows. Stream-scaled memory sinking handles KV-cache entries evicted from the 9-frame window (with an attention sink of 3): at detected semantic boundaries they are discarded, EMA-smoothed into a sink, or appended, so stale context does not poison new frames.
VLM-teacher loop: a grader that never solves the problem. The teacher VLM reads the task condition and synthesizes its own supervision: one goal-achievement query (“did the final frame satisfy the objective?”) plus M process-supervision queries, with M typically between 1 and 3. During the student video model’s denoising, the VLM answers those queries about the predicted frames as a multi-question VQA problem, and the VQA loss is differentiable, so gradients flow through the predicted frames into a small per-instance LoRA. There is no RL, no trained reward model, no policy gradient noise. Optimization early-stops when the multi-VQA loss drops below a threshold or hits a step cap, so easy instances stop early and hard ones get more compute.
Key numbers
| Axis | Stream-T1 | VLM-teacher loop | Setting and source | Same harness? |
|---|---|---|---|---|
| Intervention | training-free chunk-level search: noise propagation, reward pruning, memory sinking | per-instance LoRA tuned by differentiable VLM grader rewards | both keep the video backbone frozen | n/a |
| Headline short-clip gain | 5s VideoAlign MQ 0.350 → 0.629; VQ 0.285 → 0.426; TA 1.193 → 1.305 | RULER-Bench 46.4 → 68.2 (+21.8) | Stream-T1 vs LongLive, 946 VBench prompts; teacher vs Wan2.2-5B-Distilled frozen baseline | No |
| Headline long-clip gain | 30s VideoAlign MQ -0.002 → 0.226; VQ -0.169 → -0.073 | VBVR-Bench 0.666 → 0.781 (+0.115) | Stream-T1 on 128 MovieGen prompts, 16 FPS, 832x480; teacher on symbolic reasoning bench | No |
| Saturated VBench metrics | 5s: motion smoothness 99.12 → 99.15, subject consistency 97.00 → 97.25; 30s: aesthetic 61.56 → 62.11, subject 97.90 → 98.43, smoothness 98.78 → 99.03 | n/a | Stream-T1 only; already 97 to 99 on baseline, so near-noise | Yes, within Stream-T1 |
| Ablations | (page does not isolate the three mechanisms numerically) | VBVR: drop final-goal reward 0.692 (−0.089); drop all task rewards 0.712 (−0.069); drop process queries 0.758 (−0.023) | teacher paper, full score 0.781 | Yes, within teacher paper |
| Best-of-n baseline | implicit target: single-trajectory baseline collapses at 30s | Pass@5: 0.683 VBVR / 49.1 RULER (+0.017 / +2.7); VideoTPO: 0.634 VBVR (−0.032), +3.9 RULER | brute-force sampling vs structured-reward optimization | Yes, within teacher paper |
| Failure audit | reward-model blind spots (no audit counts on the page) | 50 audited failures: 42 (84%) VLM perception errors, 8 (16%) reward-query errors; R² = 0.733 between teacher capability and RULER score | teacher paper’s own error analysis | Yes, within teacher paper |
Read the table rows against each other and the picture is consistent: Stream-T1’s large numbers live on reward-model metrics (VideoAlign), because the VBench consistency metrics were already saturated at 97 to 99 and have almost no headroom; its 30s row is where the method earns its keep, lifting motion quality from roughly zero to 0.226. The teacher paper’s large number (+21.8 on RULER-Bench) comes with its own honest counterweight: on the symbolic VBVR-Bench the gain is +0.115, and the failure audit says the ceiling is the grader’s eyes, not the optimizer.
What each paper is really claiming
From Stream-T1: a frozen streaming model leaves real headroom on the table, and pure inference-time search reclaims it, biggest exactly where streaming models fall apart, past their reliable length. The evidence is deliberately narrow: one baseline (LongLive), two benchmarks, 832x480, up to 30 seconds. The paper also concedes the tension its own method creates: expanding and scoring multiple candidates per chunk multiplies inference compute, which sits awkwardly next to the “real-time streaming” framing.
From the VLM-teacher paper: extra samples do not create reasoning, structured rewards plus per-instance optimization do. Best-of-five sampling adds only +0.017 on VBVR-Bench and +2.7 on RULER-Bench, while the teacher loop adds +0.115 and +21.8; an alternative preference-optimization baseline (VideoTPO) actually drops VBVR to 0.634. The ablations back the mechanism story: the final-goal query is the single most important channel (−0.089 without it), and the R² = 0.733 correlation plus the 84-percent perception-error audit define the ceiling: a sharper teacher VLM would move the numbers, and nothing in the method fixes a teacher that misreads the frame.
When to use which
- Your streaming model drifts on long clips and you cannot retrain: Stream-T1. It is a wrapper, not a new generator: the noise-propagation, reward-pruning, and memory-sinking recipe sits on top of a frozen streaming model, and its biggest gains (30s MQ from -0.002 to 0.226) are precisely the drift regime.
- Your video model generates fluent motion but fails the task: the VLM-teacher loop. If the failure is “the video looks fine but does not satisfy the constraint,” search over chunks will not help; a grader that checks goal achievement and reasoning paths will.
- You need real-time output: be careful with both. Stream-T1 is framed for streaming but its search multiplies per-chunk compute; the teacher loop is explicitly per-instance inference-time training, with latency far above a single forward pass.
- You only have API-level access to the model: Stream-T1 is closer to feasible (it still needs reward model scores per candidate), while the teacher loop requires backpropagating through the video model’s denoising, which typically means owning the weights.
- You are also considering training-free sampler fixes: methods like MIGA improve long-video stability by changing how the frozen sampler runs, without per-chunk candidate expansion, a cheaper first thing to try before paying search or optimization costs.
Limits and open questions
Nobody has run these two methods under one harness, and the table above says so row by row: Stream-T1’s numbers come from a LongLive baseline on VBench and MovieGen prompts, the teacher’s from Wan2.2-5B-Distilled on RULER-Bench and VBVR-Bench. Different backbones, different resolutions, different tasks (fidelity versus reasoning), so “search versus optimization” has no shared-benchmark answer yet. Stream-T1 inherits the blind spots of its reward models (optimizing VideoAlign hard can drift toward what the scorer likes), reports against a single baseline, and leaves the search-versus-real-time-latency tension unresolved. The teacher loop inherits the grader’s perception ceiling, is demonstrated on a step-distilled 5B student, and its per-instance optimization cost is real but unquantified on the page. The open question both leave behind is the same: when test-time compute is the currency, what is the exchange rate (how much latency buys back how much quality), and neither paper prices it.
FAQ
Can I compare Stream-T1’s +0.28 MQ with the VLM-teacher’s +21.8 RULER points directly?
No. They are different metrics on different tasks against different frozen baselines: VideoAlign motion quality on 5-second streaming clips versus RULER-Bench reasoning scores on task-constrained generation. The defensible cross-paper statement is about the shape of the win, not its size: both methods move their own reward-based metrics a lot while near-saturated conventional metrics (VBench consistency in Stream-T1’s case) barely budge.
Does either Stream-T1 or the VLM-teacher loop retrain the video model?
Neither retrains the backbone. Stream-T1 is fully training-free: it searches over candidate chunks at inference. The teacher loop also freezes the video model, but it does tune a small per-instance LoRA at inference time, so it needs gradient access to the model rather than just more forward passes.
Why does the VLM-teacher loop beat best-of-five sampling so much on RULER-Bench?
Because sampling more videos explores, while structured rewards plus per-instance optimization direct. Pass@5 reaches only 49.1 on RULER-Bench versus 68.2 for the teacher loop; the paper’s read is that brute-force sampling adds diversity but no feedback, while the teacher’s goal and process queries tell the optimizer what is wrong with each specific attempt.
What actually caps each method?
Stream-T1 is capped by its reward models: it moves VideoAlign scores a lot, and it inherits whatever those reward models miss. The teacher loop is capped by the grader’s perception: 84% of its audited failures were the VLM teacher missing a subtle visual error, and RULER-Bench score correlates with teacher capability at R² = 0.733. In both cases the scorer, not the generator, is the ceiling.
Which one should I try first on a streaming workload?
If the symptom is drift on long clips, start with the cheaper training-free fixes (sampler alignment such as MIGA, then Stream-T1’s search if you can afford the per-chunk compute), because Stream-T1’s gains concentrate exactly in that regime. If the symptom is constraint violation on short tasks, the teacher loop targets that failure directly, but only if you can backpropagate through your model at inference time.