MemPrivacy vs TaskMem: What Should an AI Agent Remember?
MemPrivacy filters what may leave your device for cloud memory, holding utility loss within 1.6%. TaskMem learns what a video agent should write down at all, lifting VideoMME from 61.6% to 67.9%.
The two papers guard the same gate from opposite sides
Every long-running agent faces the same silent decision, dozens of times a day: this observation just arrived, what do we do with it? Writing everything to memory is expensive and unsafe. Writing nothing makes the agent a goldfish. The recent survey of persistent autonomous AI argues this is the real leap from chatbot to digital colleague: not a bigger model or more tools, but persistent state that survives across sessions. Two 2026 papers attack that state from opposite directions, and they pair up more naturally than they compete.
MemPrivacy (MemTensor, HONOR, Tongji; arXiv 2605.09530) looks at the write from the privacy side. When an edge device offloads memory management to a cloud agent, sensitive spans such as blood pressure readings and email addresses leave the phone in plaintext. MemPrivacy detects those spans on the device, replaces them with type-aware placeholders, lets the cloud memory system work over the placeholders, and restores the original values locally when the reply comes back. Protection is reversible by design, because for agent memory you almost always want the value back later.
TaskMem (ByteDance Seed, Fudan; arXiv 2605.31075) looks at the write from the utility side. A multimodal agent streaming egocentric video faces a combinatorial explosion of observations, and most memory pipelines write what a frozen captioner happens to produce. TaskMem reframes memory generation as a learnable policy trained with reinforcement learning, so the agent decides what is worth remembering based on the tasks its deployment environment actually rewards.
Same gate, opposite questions. MemPrivacy asks “what must not leave the device?” TaskMem asks “what deserves to be kept at all?” A system that runs one and not the other either leaks or forgets.
How MemPrivacy protects the write
The mechanism has three stages, and the order matters. First, an on-device detector tags privacy-sensitive spans in the user’s message. Second, the tagged values are replaced with typed placeholders, so “my blood pressure today was 160/110” becomes <Health_Info_1> before anything is transmitted. Third, the cloud agent or memory system does its normal work over the placeholder text, and the original values are looked up and reinserted locally before the user sees the reply. The cloud never stores the raw values, but it keeps enough typed structure to stay useful.
Two design choices carry the results. Typing the placeholders (health info, email, name) preserves the semantic category that generic masking destroys; the paper shows untyped placeholders still lose 4.72-8.71% utility, so the type annotation is doing real work. And because the detector only needs to find spans, not understand the task, a small specialist model suffices: the authors train six detectors from 0.6B to 4B parameters with supervised fine-tuning followed by GRPO where the reward is detection F1.
They also built MemPrivacy-Bench, because no existing benchmark covered graded, reversible memory privacy. The training split holds 26,016 turns from 160 users (125k+ privacy instances) and the test split 6,337 turns from 40 users (29.9k instances), about 1M dialogue tokens in total, split evenly between Chinese and English. Human verification put annotation accuracy at 98.08%. Evaluation runs against three widely used memory systems: LangMem, Mem0, and Memobase.
How TaskMem shapes the write
TaskMem treats memorization as a policy rather than a fixed summarization step. Its training splits into two phases. Phase One teaches the policy how to memorize: optimize memory quality under fundamental fidelity requirements such as correctness, non-redundancy, and format compliance, formulated as a multi-objective RL problem. Phase One alone gives a modest bump of 1 to 2 points on the benchmarks, which tells you generic memory quality is not the bottleneck.
Phase Two is the contribution. After deployment, the policy learns what to memorize by tuning a lightweight adapter of just 2,048 parameters at transformer layer 22 of the base model, using the Phase One model as the frozen reference. Sparse, noisy task feedback is converted into pairwise preference data through a task-relevance reward model, so the agent biases its memory writing toward what the deployed tasks reward, without touching base weights. Cold start is the catch: with no task signal yet, you are back to Phase One quality.
The evaluation is deliberately honest about difficulty. The authors reformulate VideoMME, EgoLife, and EgoTempo into streaming benchmarks where questions must be answered from memory alone, with no access to raw video, on a Qwen3-VL-30B-A3B backbone.
Key numbers
| Dimension | MemPrivacy | TaskMem |
|---|---|---|
| What the write is optimized for | Keep sensitive spans off the cloud while memory stays useful | Keep task-relevant observations in memory, drop the rest |
| Mechanism | On-device span detection, typed reversible placeholders, local restore | Two-phase RL memorization policy; Phase Two tunes a 2,048-parameter adapter at layer 22 |
| Benchmark scale | MemPrivacy-Bench: 200 users (160 train, 40 test), 125k+ train and 29.9k test privacy instances, about 1M dialogue tokens, Chinese and English split evenly; annotation accuracy 98.08% | VideoMME, EgoLife, EgoTempo reformulated as streaming memory-only QA |
| Headline metric | Detection F1 85.97% on MemPrivacy-Bench, 94.48% on PersonaMem-v2 | VideoMME accuracy 67.9% (P1+P2) vs 61.6% baseline; EgoLife 45.4% vs 38.4%; EgoTempo 27.6% vs 22.3% |
| vs frontier general models | +16.98 F1 over GPT-5.2, +7.56 over Gemini-3.1-Pro on MemPrivacy-Bench; +6.42 and +7.89 on PersonaMem-v2 | GPT-5.2 reference slightly edges TaskMem on EgoTempo temporal questions despite TaskMem’s higher precision |
| Cost of the mechanism | Utility loss 0.71-1.60% (full PL2-PL4 protection), 0.08-0.33% (PL4 only), across LangMem, Mem0, Memobase | Phase One adds only 1-2 points; Phase Two delivers the rest; adapter is 2,048 parameters on a 30B-class backbone |
| Baseline it beats | Irreversible masking loses 16.99-41.87% utility; untyped placeholders lose 4.72-8.71%; OpenAI-Privacy-Filter reaches only 35.50% F1 | Frozen-captioner memory pipeline on Qwen3-VL-30B-A3B: 61.6% / 38.4% / 22.3% |
| Latency and size | On-device detection about 2s on MemPrivacy-Bench, under 1s on PersonaMem-v2; detectors are 0.6B-4B | Not the focus; the Phase Two adapter is small by construction (2,048 parameters) |
Read the two columns as different measurement types, not as one leaderboard. MemPrivacy’s F1 measures span detection quality on the authors’ own benchmark, with utility measured as the downstream cost across three specific memory libraries. TaskMem’s accuracy measures question answering from memory over streaming video, where memory precision on VideoMME reaches 85.6% with 79.3% coverage, and on EgoLife climbs from 73.3% to 80.5%. The honest cross-paper claim is narrow: both papers show the memory write gate is learnable and cheap to control, in directions where the naive default (mask everything, or memorize everything) is either catastrophic (up to 41.87% utility loss) or useless (a frozen captioner that ignores the task).
When to use which
You are building or operating a memory layer for agents and want to know which paper to steal from. The answer is almost always “both, in this order”, but the priority depends on your deployment.
Choose MemPrivacy first if user text can contain personally identifiable information and any part of memory processing happens off the device. The numbers make the case plainly: full protection costs 0.71-1.60% utility, while the masking reflex costs 16.99-41.87% on the same systems. The edge-cloud split is practical, not aspirational, because a 0.6B-4B specialist running locally at about 2 seconds per interaction beats GPT-5.2 on the detection task itself (+16.98 F1).
Choose TaskMem first if your agent drowns in observations and your bottleneck is memory content quality rather than privacy. This is the profile of wearables, screen-watching assistants, and robots logging a day: a frozen captioner writes what it writes, and the deployed tasks suffer for it. A 2,048-parameter adapter tuned with DPO on task-relevance preferences is the kind of cheap knob that survives contact with production, provided you have real task feedback to build the preference pairs.
If you run both, the sequence matters: TaskMem’s policy decides what deserves a slot in memory, and MemPrivacy’s filter decides what may leave the device to get one. TaskMem assumes the memory store it writes to is trusted; MemPrivacy assumes the content written is worth protecting. Neither assumption covers the other’s problem, and neither paper evaluates the combination, which is a real gap rather than a complaint.
Limits and open questions
MemPrivacy’s guarantee is only as strong as the detector’s recall: with 85.97% F1, some sensitive spans leave the device in plaintext no matter what. The benchmark is the authors’ own, human-annotated to 98.08% accuracy, and the GPT-5.2 and Gemini-3.1-Pro comparisons are not third-party reproduced. Typed placeholders reduce but do not eliminate leakage, since the cloud still learns the structure and types of your private data. The threat model assumes an honest-but-curious cloud, not one actively trying to invert placeholders with auxiliary information, and the sub-1.6% utility figure is measured on three memory libraries, not on arbitrary multi-turn agent tasks.
TaskMem’s evaluation is entirely VQA-style: answer a question from memory. It does not show the policy helping on planning, tool use, or action selection, which is where agent memory ultimately has to prove itself. Phase Two needs real task feedback, so cold start falls back to Phase One quality. The quality reward leans on GPT-4o and Gemini-2.5-Flash as judges, inheriting their biases and cost. And the absolute EgoTempo numbers (sub-30% across all methods) are a reminder that temporal reasoning over egocentric streams is far from solved; a GPT-5.2 reference slightly beats TaskMem there despite its higher precision.
FAQ
How do MemPrivacy and TaskMem differ?
MemPrivacy controls which memory content may leave the device, using typed reversible placeholders processed locally. TaskMem controls which observations deserve to be written to memory at all, using a reinforcement-learned memorization policy. One is a privacy filter on the write, the other is a relevance filter.
What results does MemPrivacy report on MemPrivacy-Bench?
The best MemPrivacy detector reaches 85.97% F1, which is +16.98 over GPT-5.2 and +7.56 over Gemini-3.1-Pro, while the OpenAI-Privacy-Filter baseline manages only 35.50%. Full PL2-PL4 protection costs 0.71-1.60% task utility across LangMem, Mem0, and Memobase, versus 16.99-41.87% for irreversible masking.
What results does TaskMem report on VideoMME and EgoLife?
TaskMem (Phase One plus Phase Two) reaches 67.9% accuracy on VideoMME versus a 61.6% baseline, 45.4% on EgoLife versus 38.4%, and 27.6% on EgoTempo versus 22.3%. Phase One alone adds only 1-2 points; the 2,048-parameter Phase Two adapter delivers the rest.
Can the two methods be combined?
They solve different problems and compose naturally: TaskMem decides what is worth remembering, MemPrivacy decides what may go to the cloud. But neither paper evaluates the combination, so the joint cost on utility, latency, and memory quality is an open question for builders.
Are the numbers reproducible outside the papers’ own setups?
Both papers use their own benchmarks (MemPrivacy-Bench; streaming reformulations of VideoMME, EgoLife, and EgoTempo), so treat the headline figures as protocol-bound. MemPrivacy releases code and models, TaskMem points to a project page, but third-party reproduction on your own tasks is the check that matters.