MemDreamer vs TaskMem: Learn What to Remember, or Just Retrieve Better?

The one design split

Both papers attack the same wall: a two-hour video sampled at 1 FPS is over 1.6M tokens, past every current context window, so a coupled VLM either subsamples frames into noise or drowns in attention dilution and lost-in-the-middle failures. Both refuse to retrain the base model. That is where the agreement ends, because they place the bottleneck at opposite ends of the memory pipeline.

MemDreamer treats memory as a read-side problem. Assume a strong perception model watches the video once, offline, and writes a compact structured store; the engineering risk is whether a reasoning model can find the right fact at question time. Its answer is a three-tier hierarchical graph (Video Root, Super Events, Macro Events with entity and micro-event subgraphs) queried by seven tools in an iterative loop, not by embedding similarity.

TaskMem treats memory as a write-side problem. Assume the agent watches an unbounded live stream and will never see the raw video again; the engineering risk is that a fixed captioner writes the wrong things: too little and the answer is gone, too much and the memory drowns in irrelevant detail. Its answer is to make memory writing a policy trained with RL, then adapt it post-deployment with a preference-tuned adapter small enough to ship per customer.

Neither paper does both. MemDreamer’s graph is written by a perception model doing ordinary captioning, not a learned write policy. TaskMem’s agent answers from its written memory with ordinary reading, not structured retrieval. That is what makes the comparison useful: they are two halves of the same system, and the numbers below tell you what each half is worth on its own.

Key numbers

Measurement	MemDreamer	TaskMem	Setting and source	Same harness?
Base model, frozen	Gemini-3.1-Pro (perception + reasoning)	Qwen3-VL-30B-A3B	Both papers keep the backbone frozen	No (different backbones)
Training	None (training-free)	GSPO RL over 326 videos, then DPO on a 2,048-parameter adapter at layer 22	TaskMem paper	n/a
LVBench	90.7 vs 78.2 end-to-end (+12.5)	Not run	MemDreamer paper	Yes, same Gemini-3.1-Pro
LongVideoBench	92.9 (+14.3 vs end-to-end)	Not run	MemDreamer paper	Yes, same model
Video-MME (long)	91.0 (+14.0 vs end-to-end)	67.9 vs 61.6 baseline (+6.3)	Both papers use this benchmark, but different backbones	No; do not compare 91.0 with 67.9 directly
EgoSchema	92.1 (+11.8 vs end-to-end)	Not run	MemDreamer paper	Yes, same model
EgoLife	Not run	45.4 vs 38.4 (+7.0)	TaskMem paper	Yes, same Qwen3-VL-30B
EgoTempo	Not run	27.6 vs 22.3 (+5.3)	TaskMem paper	Yes, same backbone
Reasoning context per run	5.9K-6.3K tokens vs 240K-784K end-to-end (about 40x cut)	Not reported in tokens	MemDreamer paper	Yes, same-model comparison
Retrieval method ablation	Embedding lookup 70.5 vs full agentic toolkit 90.7 (+20.2)	n/a	MemDreamer paper	Yes
Graph structure ablation	Flat store 77.4; +causal/temporal edges 84.8; +hierarchy 86.3; full graph 90.7	n/a	MemDreamer paper	Yes
Phase split (VideoMME)	n/a	Phase One 64.4, Phase One + Two 67.9	TaskMem paper	Yes
Memory coverage / precision (VideoMME)	n/a	79.3% / 85.6%; EgoLife precision 73.3% -> 80.5%	TaskMem paper	Yes

Read the table for what it does not contain. There is no row where both systems run the same benchmark with the same backbone: Video-MME appears in both papers, but as 91.0 on Gemini-3.1-Pro against 67.9 on Qwen3-VL-30B-A3B, a backbone gap, not a memory-method gap. So this comparison can rank each method against its own end-to-end baseline, and it can size each design choice with ablations, but it cannot tell you which system would win head-to-head.

What MemDreamer’s read side buys

The headline result is a same-model comparison, which is the clean kind: Gemini-3.1-Pro run end-to-end over LVBench scores 78.2, and the same weights inside the MemDreamer harness score 90.7, a +12.5 lift that leaves the system 3.7 points behind the human-expert reference. The gain replicates across LongVideoBench (92.9, +14.3), Video-MME long (91.0, +14.0), and EgoSchema (92.1, +11.8). Because nothing is trained, the harness helps weaker reasoners even more: Qwen3-VL-235B-A22B-Thinking jumps from 63.6 end-to-end to 84.8, a +21.2 gain.

The ablations are the argument. Strip the graph back to a flat chunk store and LVBench falls to 77.4, barely better than end-to-end; add only the causal and temporal edges and it recovers to 84.8; add only the hierarchy and it recovers to 86.3; the full structure reaches 90.7. Hierarchy and edges fix different failures and compound. On the retrieval side, a static embedding lookup scores 70.5 while the seven-tool agentic loop scores 90.7, a 20.2-point gap, with graph traversal along entity and causal edges the single biggest tool contribution at +6.6. Dumping the entire graph into context (“Full Memory”) helps summarization (81.3) but collapses on reasoning questions (72.7), because unfiltered structure adds topological noise. This is the same lesson as rethinking retrieval beyond semantic similarity: for agentic search, a similar clip is not a relevant clip.

The most interesting number is a correlation. Across eight models, AIME2025 reasoning scores correlate with LVBench at Pearson 0.702 (p = 0.052, not significant) when models ingest video end-to-end, but at 0.897 (p < 0.01) inside MemDreamer. Raw video tokens block a model from using the reasoning ability it already has; decoupling restores the link. That reframes long-video progress as a retrieval-and-memory problem more than a perception problem, and it echoes the decoupling instinct in Watch, Remember, Reason.

What TaskMem’s write side buys

TaskMem’s headline is smaller in absolute terms but answers a harder constraint: the agent must answer from memory alone, in a streaming setting where re-watching is impossible. Reformulating three benchmarks into that protocol, the trained write policy reaches 67.9% on VideoMME, 45.4% on EgoLife, and 27.6% on EgoTempo, beating the Qwen3-VL-30B-A3B baseline by 6.3, 7.0, and 5.3 points. The largest absolute gain is on egocentric daily-life video, exactly the regime where indiscriminate memorization fails.

The phase split is the evidence that task-shaping, not generic summarization quality, does the work. Phase One (GSPO RL over 326 long videos with a multi-objective reward combining format compliance, a soft length penalty, a GPT-4o/Gemini-2.5-Flash-judged quality term, and within-group richness ranking) lifts VideoMME from 61.6 to 64.4, a modest bump. Phase Two converts sparse deployment feedback into DPO pairs through a task-relevance reward model and tunes only a 2,048-parameter adapter at transformer layer 22, with the 30B base frozen as the reference. That phase delivers the rest of the gain, to 67.9, while pushing memory coverage to 79.3% and precision to 85.6% on VideoMME and EgoLife precision from 73.3% to 80.5%. Adapting what an agent remembers becomes a knob you ship per deployment instead of a retraining project.

Two honesty markers stand out. On EgoTempo, all methods stay under 30%, and a GPT-5.2 reference slightly edges TaskMem on temporal questions despite TaskMem’s higher precision; temporal reasoning over egocentric streams is not solved. And the whole reward stack leans on frontier judges, inheriting their biases and their per-call cost.

When to use which

• Recorded video, questions arrive later, offline compute is affordable: MemDreamer. One perception pass builds the graph, every later question is cheap, and the +12.5 to +14.3 same-model lifts come with zero training.
• Live or unbounded streams, no chance to re-index: TaskMem. MemDreamer’s architecture assumes an offline construction pass; a wearable or screen-watching agent that must answer from what it already wrote down needs a write policy, and the 2K-adapter design is built for post-deployment adaptation.
• Your bottleneck is finding facts in a good store: retrieval is the gap, so graph structure and agentic tools are the lever: the 20.2-point embedding-to-agentic gap is the biggest single number in this comparison.
• Your bottleneck is the store itself containing the wrong facts: writing is the gap, so a task-shaped write policy is the lever: Phase Two’s adapter-only gain on top of Phase One shows task relevance is what moves the score.
• Weaker open-source backbone: both help more when the base model is weaker. MemDreamer gains +21.2 on Qwen3-VL-235B against +12.5 on Gemini; TaskMem’s entire premise is making a frozen 30B useful in a regime where end-to-end reading fails.

Limits and open questions

MemDreamer’s costs are hidden upstream. The “about 40x context cut” describes inference only: the reasoning window holds 5.9K-6.3K tokens, but building the memory consumed a 40.3K-token perception pass per video on LVBench, and perception errors propagate silently; an entire wrong subgraph caps accuracy and the agent cannot recover what perception never wrote. Swapping the perception model between Gemini-2.5-Pro and Gemini-3.1-Pro moves the final score only 0.4-1.4 points, which is reassuring, but no wall-clock or dollar figure is reported for either phase. The 12-round tool budget, search top-k, and embedding model are all tuned knobs, and every benchmark is curated video QA; open-ended summarization at scale and live-updating streams are untested.

TaskMem’s costs are hidden downstream. Phase Two needs real task feedback from the deployment environment to build preference pairs; in a cold start with no task signal you are back to Phase One quality, 64.4 instead of 67.9. The evaluation is entirely VQA-style, so the paper shows a memory-writing policy improving question answering, not planning, tool use, or action selection, which is where agent memory ultimately has to prove itself. And sub-30% EgoTempo scores across all methods are a reminder that even the best write policy cannot rescue temporal reasoning the backbone cannot do.

The obvious missing experiment is the combination: TaskMem’s trained write policy feeding MemDreamer’s graph read side, evaluated on a shared benchmark with a shared backbone. Neither paper runs it, and until someone does, “read side or write side matters more” has no controlled answer.

FAQ

Is MemDreamer or TaskMem better?

There is no head-to-head number. MemDreamer’s strongest evidence is same-model: Gemini-3.1-Pro goes from 78.2 to 90.7 on LVBench inside its harness. TaskMem’s strongest evidence is baseline-relative: Qwen3-VL-30B-A3B goes from 61.6 to 67.9 on VideoMME with a trained write policy. The Video-MME numbers from the two papers (91.0 vs 67.9) sit on different backbones and cannot be compared directly.

Do MemDreamer or TaskMem retrain the base VLM?

No, and that is the shared premise. MemDreamer is training-free: the same frozen VLM acts as perception and reasoning. TaskMem freezes the 30B Qwen3-VL backbone and trains only a 2,048-parameter adapter at layer 22 with DPO, using the Phase One model as the reference.

How much context does MemDreamer actually save?

The reasoning model holds about 5.9K-6.3K tokens per run instead of the 240K-784K an end-to-end VLM ingests, roughly a 40x cut. That figure covers the reasoning phase only; the offline perception pass that builds the graph used 40.3K tokens of perception context per video on LVBench.

Why does agentic retrieval beat embedding search in MemDreamer?

A static embedding lookup scores 70.5 on LVBench against 90.7 for the full seven-tool loop, a 20.2-point gap. Single-turn semantic recall finds visually similar clips with no causal link to the question and cannot self-correct, while the iterative loop follows entity and causal edges; graph traversal alone contributes +6.6.

What does TaskMem’s two-phase training buy over RL alone?

Phase One (GSPO RL for memory quality) lifts VideoMME from 61.6 to 64.4. Adding Phase Two (DPO on a 2,048-parameter adapter tuned with task-relevance preferences) reaches 67.9, so most of the gain comes from task shaping, not from better generic summarization. Across all three benchmarks Phase One contributes only 1-2 points and Phase Two the rest.

Can you combine MemDreamer and TaskMem?

Conceptually they are complementary halves: a learned write policy producing better graph nodes, agentic retrieval reading them. No published result runs that combination, and it is the most interesting open experiment in this space.