InternVideo3: An 8B Video Agent with Contextual Reasoning

Quick answer

InternVideo3 is an 8B video-language model from Shanghai AI Lab and Nanjing University that does two separable things. As a plain model it scores 73.8 on Video-MME, 77.3 on MLVU, and 76.6 on EgoSchema, beating its 7B predecessor InternVideo2.5 (65.1 / 72.8 / 63.9) and the similarly sized Qwen3-VL-8B (71.4 / 57.6 / 69.8). Separately, when you wrap that model in an agentic loop the authors call Multimodal Contextual Reasoning (MCR), Video-MME climbs from 73.1 to 75.8, a +2.7 from inference-time tool use rather than from the weights. The third piece is plumbing: a custom attention variant (M2LA) that runs 768K tokens on a single H200 where the base architecture runs out of memory at 512K.

Keep those three apart. The single-pass score is what the 8B weights can do. The +2.7 is what an agent harness with tools buys on top. The 768K context is what makes minute-scale video feasible at all.

The problem: long video breaks one-pass QA

A 15-minute video at 4 fps is roughly 256K tokens. Two things fail at that length. First, you cannot fit the KV cache: the base Qwen3-8B attention OOMs around 512K tokens on an H200, so true long video is off the table. Second, answering a hard temporal question in one forward pass means the model has to find the relevant moment, read it, and reason about it all at once, with no chance to go back and look again. InternVideo3 attacks both: M2LA fixes the memory wall, and MCR turns one-pass QA into a look-reason-act-look-again loop.

How the agentic loop works, and what it is not

MCR treats a question as a closed loop over a shared, growing context. Each round the model observes visual tokens, writes down intermediate reasoning, picks an action, and folds the result back in. The actions are real tools: video segmentation, ASR, temporal grounding, web search, memory read/write, a verification step, and a termination step. On top sits a video-agent scaffold with hierarchical memory (frame index, scene boundaries, clip captions, timestamps), a router that classifies the question type first, and a recursive self-check before the model commits to an answer.

Here is the line to hold. The +2.7 on Video-MME comes from this harness, not from the base weights getting smarter at test time. The agent can re-query a segment, run ASR on a span it misheard, or verify a claim before answering, and those extra passes are where the gain lives. If you report 75.8 as “the model’s score” you have quietly credited the search budget and the tool stack to the 8B network. The honest framing: 73.1 is the model, 75.8 is the model plus an agent loop with a tool budget. This is the same separation that matters in search-agent harnesses, where the scaffold and the policy contribute distinct slices of the final number.

M2LA: how it fits 768K tokens without dropping any

M2LA (Multimodal Multi-head Latent Attention) compresses the KV cache instead of pruning tokens. It factors keys and values into a compact latent space and reconstructs them per head, adds modality-aware adapters so text and vision get separate affine maps on the cached latents, and budgets compression more aggressively in deeper layers. The trick that makes it work on a Qwen3 backbone is handling QK-Norm: the paper observes that the RMSNorm coefficients concentrate in high-dimensional space and approximates the norm as a constant linear scaling, which lets the latent factorization survive the norm layer.

The payoff is on a single H200: 1.84x decode speedup at 32K tokens, 4.12x at 128K, and up to 5.01x at 256K to 384K. The base model OOMs at 512K; M2LA runs to 768K. The same low-rank-KV idea shows up in VideoMLA’s latent KV cache, which ports the trick into video diffusion rather than understanding, so this is a converging direction across video systems, not a one-off.

How it was trained

Four stages. Continued pre-training on 16M multimodal samples (about 13.5B tokens) to recover language and alignment after the M2LA conversion. Short-to-long supervised fine-tuning, first at 2 fps up to 512 frames, then 4 fps up to 2048 frames, on 379K long videos averaging 15.8 minutes with over 1M QA pairs. Rule-based RL with GSPO, using IoU rewards for grounding and correctness rewards for multiple choice. Then on-policy distillation from a Qwen3-235B teacher with a reverse-KL loss. The ablation names continued pre-training as the single biggest lever: dropping it costs 2.1 points on average (66.8 to 64.7).

Key results

• Video-MME (long video, 8B): 73.8 vs InternVideo2.5-7B 65.1 and Qwen3-VL-8B 71.4. Proprietary frontier is still ahead, with Gemini 3 Pro at 88.6, so this is a strong open 8B, not a frontier model.
• EgoSchema: 76.6 vs Qwen3-VL-8B 69.8 and GPT-5 75.6, the one benchmark where the 8B edges a named frontier system.
• MLVU / LongVideoBench / VRBench: 77.3 / 66.8 / 69.4, each ahead of both InternVideo2.5 and Qwen3-VL-8B; VRBench is the widest open-model gap (69.4 vs 51.9 for InternVideo2.5).
• Agentic inference (MCR at test time): Video-MME 73.1 to 75.8, a +2.7 from the tool loop, separate from the weights.
• M2LA efficiency: 4.12x decode speedup at 128K tokens on one H200, and 768K context where the base OOMs at 512K.
• Short video, honest mix: edges its predecessor on NextQA (85.5 vs InternVideo2.5’s 84.9) and tops Qwen3-VL-8B on VSIBench (68.1 vs 59.1), but loses MVBench to that same predecessor (75.0 vs 75.7), so the gains are concentrated in long and spatiotemporal tasks, not uniform.

Limits and open questions

The paper is candid that newer models (Qwen3.5+, GLM, Kimi) already surpass InternVideo3, so the comparison set is a moving target. M2LA is tied to converting an existing GQA backbone and does not apply cleanly to models that are already native MLA or linear-attention. MCR is implicit state modeling, not a full action-conditioned world model, and tool errors propagate through the loop with limited joint optimization to catch them. Evaluation is video-only: there is no evidence for GUI, mobile, or embodied agent settings despite the “agentify foundation models” framing. And the agentic gain is reported on Video-MME; whether +2.7 holds across the other long-video benchmarks, or shrinks, is not shown.

FAQ

What is InternVideo3 and how is it different from InternVideo2.5?

InternVideo3 is an 8B video-language model from Shanghai AI Lab and Nanjing University. Versus the 7B InternVideo2.5 it jumps Video-MME 65.1 to 73.8, MLVU 72.8 to 77.3, and EgoSchema 63.9 to 76.6. The two structural additions are M2LA attention for long context and an agentic MCR loop for inference-time tool use, neither of which existed in 2.5.

Does InternVideo3’s Video-MME score come from the model or from the agent loop?

Both, and they are reported separately. The 8B weights score 73.1 to 73.8 single-pass. The agentic MCR loop, which can re-query segments, run ASR, ground timestamps, and self-verify, adds +2.7 on top (73.1 to 75.8 on Video-MME). Crediting the full 75.8 to the model alone would attribute the tool budget and search to the base network, which the ablation does not support.

How does InternVideo3 fit 768K tokens on a single GPU?

Through M2LA (Multimodal Multi-head Latent Attention), which compresses the KV cache into a low-rank latent space and reconstructs per head, with modality-aware adapters and deeper-layer budgeting. On one H200 it gives a 4.12x decode speedup at 128K tokens and runs up to 768K, where the unmodified Qwen3-8B attention OOMs at 512K.

Can InternVideo3 beat proprietary models like Gemini 3 Pro on video?

Mostly no. On Video-MME, Gemini 3 Pro scores 88.6 against InternVideo3’s 73.8. The one exception in the paper is EgoSchema, where InternVideo3’s 76.6 edges GPT-5’s 75.6. The fair claim is that it is a strong open 8B that leads similarly sized open models, not that it matches frontier proprietary systems.

One line: InternVideo3 is an 8B video model that earns 73.8 on Video-MME from its weights, adds +2.7 from an agentic tool loop, and uses M2LA to run 768K tokens on one H200. Read the original paper on arXiv.