Rethinking RAG in Long Videos: V-RAGBench and CARVE Explained

Quick answer

CARVE retrieves over long egocentric video by running four retrievers in parallel, one per modality-granularity config, then letting each chunk enter the generator under whichever config retrieved it best. On V-RAGBench, a new leak-filtered benchmark of 2,100 query/evidence/answer triplets over 216 videos (11 to 99 hours), CARVE reaches 0.603 Recall@5 against 0.510 for the strongest baseline (VideoRAG-A), and 0.357 generation pass rate with Qwen3-VL-8B against 0.315. The headline finding behind the method: the best retrieval config is a property of the chunk’s content, not of the query, so the common practice of fixing one config per query leaves recall on the table. The benchmark contribution matters as much as the method, because it first proves that over half of widely used video QA can be answered without the video at all.

The leakage problem V-RAGBench fixes

Existing video RAG benchmarks let a model score well without retrieving anything useful. The authors show that more than half of samples in common video QA sets are answerable from language priors, world knowledge, or static cues, so high generation accuracy never proved the retriever found the right moment. That breaks the entire point of evaluating retrieval.

V-RAGBench is built to close the gap. It draws 174 videos from Ego4D (averaging 86 minutes) and 42 from EgoLife (averaging 379 minutes), then passes candidate queries through five sequential filters: semantic-similarity removal, answerability verification with GPT-5.2-chat, shortcut-bias elimination (anything answerable without visual context is dropped), empirical answerability confirmation, and an evidence-uniqueness check that discards a query if its answer can be recovered from non-evidence chunks. What survives is 2,100 triplets, split 1,800 train and 300 test, each with a labeled evidence chunk. Because evidence is labeled, retrieval and generation are scored separately: Recall@5 and nDCG@5 for retrieval, an LLM-judged pass rate for generation. This decoupling is the design that makes a retrieval number trustworthy. The same instinct that retrieval needs honest, retrieval-specific evaluation runs through rethinking retrieval beyond semantic similarity.

How CARVE picks a config per chunk

A config here is one cell of a 2x2 grid: modality (visual embeddings versus textual summaries) crossed with granularity (5 keyframes chosen by k-means versus the whole 2-minute clip). Prior methods commit to one cell, or a fixed fusion of all four, for every chunk a query touches. CARVE treats that as the wrong unit of decision.

Stage 1 runs all four retrievers independently and merges their top-k results into one candidate pool, where every chunk carries a tag for the config that surfaced it. Stage 2 reranks with a multimodal cross-encoder (Qwen3-VL-Reranker-2B) that scores each chunk only under its own tag, not under all four configs, and the highest-scoring tag becomes that chunk’s winner. At generation time the evidence is fed in modality-interleaved form: one chunk arrives as keyframes, the next as a text summary, the next as a full clip, each under the representation that retrieved it. The generator therefore sees a heterogeneous evidence set rather than one forced format. This is a sharper version of the retrieval-is-not-similarity lesson that agentic memory retrieval hits from the video-understanding side.

What the per-chunk config actually buys

The ablation isolates where the 0.603 comes from. The best single config, visual embeddings on full clips, reaches only 0.507 Recall@5; the best two-way partial combination reaches 0.567; using all four configs with per-chunk selection reaches 0.603. So roughly half the gain over a single config is the modality-granularity diversity itself, and the rest is selecting per chunk rather than per query.

The reranking strategy is load-bearing too. Scoring each chunk under its own retrieving config gives 0.603, while randomly assigning a config or concatenating all configs into a joint score both drop to 0.513. That 9-point gap is the cost of ignoring which view actually found the chunk. A config-distribution table confirms there is no shortcut: winners spread across the grid (text-frame 25 to 37 percent, visual-clip 24 to 36 percent, the rest 11 to 28 percent across ranks), so no single config quietly dominates and the per-chunk machinery earns its place.

Key results

• Retrieval Recall@5: 0.603 for CARVE vs 0.510 for VideoRAG-A, the best of 8 baselines, on V-RAGBench test (k=5), a 0.093 absolute gain that proves chunk-level config beats query-level config.
• Retrieval nDCG@5: 0.433 vs 0.340 for GQR, the best baseline on this metric, confirming the ranking quality and not just hit rate improves.
• Generation pass rate: 0.357 with Qwen3-VL-8B vs 0.315 (VideoRAG-B), 0.367 vs 0.317 with Qwen3-VL-32B, and 0.320 vs 0.307 (VideoRAG-A) with Gemma-4-26B, so the retrieval gain carries through three different generators.
• Beats query-level routing without training: CARVE scores 0.357 pass rate against a trained non-LLM router at 0.329 and a LoRA-tuned LLM router at 0.310, so a per-chunk rule with no training outperforms learned per-query routing.
• Reranking matters: per-config reranking gives 0.603 Recall@5 vs 0.513 for both random config assignment and joint concatenation, a 0.090 drop when the config tag is ignored.
• Benchmark leakage: over half of samples in widely used video QA sets are answerable without the video, which V-RAGBench removes through five filters down to 2,100 evidence-labeled triplets.

Limits and open questions

The numbers are low in absolute terms. A 0.603 Recall@5 means the right evidence misses the top 5 about 40 percent of the time, and a generation pass rate near 0.36 leaves most questions unanswered, so V-RAGBench is hard and CARVE wins a hard benchmark rather than solving it. The benchmark scope is content-based and chunk-based: evidence lives in non-overlapping 2-minute chunks, which caps queries whose evidence straddles a boundary or spans long ranges, and the paper sets aside knowledge-graph or metadata retrieval entirely. Running four retrievers plus a 2B reranker for every query is more compute than a single-config pipeline, and the paper frames quality gains without a matching latency or cost budget, so the practical tradeoff against a cheaper fixed config is unquantified. The pipeline also leans on specific models (GPT-5.2-chat for filtering, Qwen3-VL components for reranking and judging), and how much the leakage filters and pass-rate judgments depend on those exact judges is not stressed. Generalization beyond egocentric Ego4D/EgoLife footage to third-person or instructional video is untested.

FAQ

Why are over half of video QA samples answerable without the video in V-RAGBench’s analysis?

The authors find that common video QA queries lean on language priors, world knowledge, or static cues rather than temporal visual evidence, so a model can answer from text alone. That inflates generation accuracy while hiding retrieval failure, which is why V-RAGBench applies five filters, including shortcut-bias elimination and an evidence-uniqueness check, before keeping its 2,100 triplets.

How does CARVE beat query-level routing on V-RAGBench without any training?

CARVE assigns a retrieval config per chunk instead of per query, then reranks each chunk only under the config that retrieved it. It scores 0.357 generation pass rate, above a trained non-LLM router at 0.329 and a LoRA-tuned LLM router at 0.310. The gain comes from treating the best config as a property of chunk content rather than a single decision made once per query.

What is CARVE’s 0.603 Recall@5 measured against on V-RAGBench?

Against 8 VideoRAG baselines at k=5 on the 300-sample test split. The strongest, VideoRAG-A, reaches 0.510 Recall@5, so CARVE’s 0.603 is a 0.093 absolute gain. The ablation shows the best single config reaches only 0.507 and the best partial two-config combination 0.567, so both modality-granularity diversity and per-chunk selection contribute.

What does modality-interleaved generation mean in CARVE?

Each evidence chunk is fed to the generator under the config that retrieved it, so one chunk arrives as 5 keyframes, another as a text summary, another as a full 2-minute clip. The generator sees a mixed-format evidence set rather than one forced representation, which is what lets per-chunk config selection carry into the generation stage.

One line: CARVE shows the right retrieval format is a property of each video chunk, not the query, and V-RAGBench shows you cannot measure that until you strip out the video QA that needed no video. Read the original paper on arXiv.