Multimodal Models · Vision Foundation Models · Alignment
MM-OCEAN vs Robust-U1: Diagnose MLLM Hallucination or Repair the Pixels
MM-OCEAN measures how often MLLMs answer correctly for the wrong visual reasons; Robust-U1 trains them to recover corrupted pixels before answering. One is a diagnostic, the other a repair.
The shared problem: right answer, wrong perception
Two failure modes dominate the reliability debate for multimodal LLMs. The first is prejudice: the model answers correctly but for reasons that have nothing to do with the image it was shown. The second is blindness under degradation: the image is corrupted, and the model reasons confidently over pixels that are not there. Both papers on this page attack the same root cause, an MLLM’s answer can look fine while its perception is not, from opposite directions.
MM-OCEAN is a benchmark. It does not try to fix anything; it measures how often correct answers rest on wrong evidence, with a per-failure-mode decomposition that a single accuracy number would hide. Robust-U1 is a training method. It does not measure grounding; it teaches the model to reconstruct corrupted visual content and then reason over both the corrupted input and its own recovered version. Diagnose versus repair. The interesting question is which one your system needs.
What MM-OCEAN measures
The benchmark formalizes Grounded Personality Reasoning (GPR), which splits one judgment into three outputs a model must produce together: a Big Five personality rating (T1), an open explanation citing observable behavior (T2), and structured multiple-choice grounding questions that pin the judgment to concrete cues such as micro-expressions, spatial location, and temporal-causal chains (T3). A right rating supported by the wrong cues counts as failure. The benchmark itself has 1,104 videos from ChaLearn First Impressions V2, 5,320 multiple-choice questions, and roughly 13,500 human-verified behavioral observations, with a text-leakage filter that strips anything answerable from the transcript alone and 77% inter-annotator agreement on verdict labels.
MM-OCEAN reports four diagnostic rates. Prejudice Rate (PR) is the headline: right rating, wrong cues. Confabulation Rate (CR) is a plausible rationale built on inaccurate evidence. Integration-failure Rate (IR) is right cues but the wrong conclusion. Holistic-Grounding Rate (HR) is the only success state, where all three outputs are consistent. This decomposition is the contribution: rating accuracy and genuine perception turn out to be almost decoupled.
What Robust-U1 does
Robust-U1 argues that handling degraded inputs should not stop at saying “the image is corrupted.” Its pipeline has three stages: supervised fine-tuning gives the model an initial corrupted-to-clean mapping; reinforcement learning with a pixel-level SSIM reward plus a semantic CLIP-similarity reward sharpens the recovered image; and the model learns to answer using both the corrupted input and the recovered version. The design targets a failure mode that pure textual reasoning about corruption cannot fix, namely missing pixel evidence, while staying more inspectable than feature-alignment methods. The paper’s driving case study is a degraded driving image where existing models infer the wrong direction and Robust-U1’s recovery supports the correct answer.
Key numbers
The two papers never share a benchmark, a task, or a harness, so nothing in this table is a head-to-head result. Read each row as what the paper itself reports, and notice how different the failure surfaces are.
| Measurement | MM-OCEAN (diagnose) | Robust-U1 (repair) | Setting and source |
|---|---|---|---|
| Headline failure signal | 51.3% mean Prejudice Rate across 27 models | R-Bench overall 0.7398 vs BAGEL 0.5770 and Robust-R1 0.5017 | Both papers’ own evaluations |
| Success rate at full rigor | 10.4% mean Holistic-Grounding Rate; best (Gemini 3 Flash) 33.5% HR with 17.2% PR | MMMB at 100% synthetic corruption: 83.18 vs BAGEL 78.48 and Robust-R1 75.35 | No shared metric |
| Weakest measured category | Spatial localization 30.7% and micro-expression detection 34.6% mean accuracy; temporal-causal reasoning 64.8% | R-Bench high-degradation CAP: 0.7640 vs BAGEL 0.4288 and Robust-R1 0.3484 | Perceptual vs degradation-specific |
| Open-vs-closed or ablation gap | Closed models beat open by 5.6 points on rating (T1) but 26.6 points on grounding (T3); best open model Qwen3.5-397B at 15.9% HR with 41.5% PR | Removing multimodal reasoning drops R-Bench 0.7398 to 0.6623; removing either reward leaves about 0.726 | Both from ablations within each paper |
| Robustness under stress | 27 models span 13 closed and 14 open; the 0-to-33.5% HR range shows grounding is nowhere saturated | Clean-to-100%-corruption drop on MMMB is 1.57 points for Robust-U1 vs 3.44 for BAGEL and 6.06 for Robust-R1 | Different stress definitions |
Three readings matter. First, MM-OCEAN’s closed-versus-open gap of 5.6 points on rating against 26.6 points on grounding says the expensive part of perception is locating evidence, not emitting a plausible score, which is exactly the capability a recovery pipeline leans on. Second, Robust-U1’s ablation shows the recovery only pays when the model actually reasons over the recovered image: without multimodal reasoning the score falls from 0.7398 to 0.6623, barely above BAGEL. Third, both papers’ hardest categories are perceptual, which suggests measurement and repair are attacking the same underlying weakness from two sides.
Diagnose versus repair, in practice
MM-OCEAN’s lesson is that accuracy hides the mechanism. A hiring-screen or ad-targeting system that scores Big Five traits can be right on average while grounding almost nothing, which is a fairness problem rather than just an accuracy one. But the benchmark is deliberately diagnostic: it proposes no training method to close the prejudice gap. Robust-U1 is the mirror image. It demonstrates a real gain on corrupted inputs, with image quality and answer quality moving together, and its own pages note the risk that a recovered image becomes a confident but wrong witness when the corruption destroyed evidence that no reconstruction can recover.
The two also fail in compositionally different places. MM-OCEAN’s integration-failure rate captures models that perceive fine and still draw the wrong conclusion; recovery cannot fix that. Robust-U1’s failure mode is the opposite: the reasoning is sound but the input evidence is missing or invented. If you stacked them, the natural pipeline would be recovery first, then grounding measurement on the recovered image, because Robust-U1 changes the evidence a judge like MM-OCEAN would be evaluating. Neither paper runs that experiment.
When to use which
- You are deploying judgment in high-stakes subjective domains such as hiring screens, mediation assistants, or personality-adjacent personalization: adopt MM-OCEAN’s decomposition. A single score will not tell you whether the model examined the evidence or matched a stereotype, and its four-rate split is the cheapest way to find out.
- Your inputs are predictably degraded (dashcams, compressed streams, low-light sensors): Robust-U1’s three-stage pipeline is the directly actionable design, and its dual SSIM-plus-CLIP reward is a small, copyable idea.
- You are building an evaluation, not a product: borrow MM-OCEAN’s structural tricks, including per-failure-mode rates, transcript-leakage filtering, and human-verified cue anchors, before you borrow its task. Apparent personality in 15-second clips is a narrow proxy.
- You are deciding whether recovery is safe for your use case: rerun the paper’s own checks locally. The method is strongest exactly when corruption is invertible; for safety-critical details, keep uncertainty checks around what the restored image asserts.
Limits and open questions
Neither paper covers the other’s ground, and the gaps compound. MM-OCEAN measures apparent personality in short, single-speaker, English clips judged partly by an AI evaluator (GPT-4o-mini), so cross-cultural and long-form perception are untested; and it shows the prejudice gap without a prescription for closing it. Robust-U1 is evaluated mainly on R-Bench, the authors’ own benchmark, with MMMB as the external check, which is better than nothing, but external replication on independent harnesses is still missing, and the recovered-image-as-evidence risk is asserted more than measured. The most obvious untried experiment is the combination: does recovery-then-answer lower the prejudice rate, or does it just make confabulated rationales look better-supported? Until someone runs that, the two papers are complements, not competitors.
FAQ
Which is better at fighting MLLM hallucination, MM-OCEAN or Robust-U1?
They do different jobs, so “better” does not apply. MM-OCEAN is a benchmark: it proves that 51.3% of correct personality ratings across 27 models rest on wrong cues, but proposes no fix. Robust-U1 is a method: it reports 0.7398 on R-Bench versus 0.5770 for BAGEL, but does not measure whether its reasoning is grounded in true evidence. Use the first to know you have a problem, the second to patch one specific cause.
Can Robust-U1’s R-Bench score be compared directly with MM-OCEAN’s grounding rates?
No. The two share no benchmark, task, model set, or grading protocol; one scores recovery-augmented answering under synthetic corruption, the other scores evidence grounding on personality video. Any direct numeric comparison would be meaningless, which is why the table above labels every row’s setting and source.
Does MM-OCEAN prove that closed models perceive better than open ones?
Partially. Closed models beat open ones by only 5.6 points on rating (T1) but 26.6 points on grounding (T3), so the closed advantage is mostly about locating evidence, not producing plausible scores. But the best open model still grounds only 15.9% fully with a 41.5% prejudice rate, and even the closed leader (Gemini 3 Flash) reaches just 33.5% holistic grounding. Nobody perceives reliably by this standard yet.
Is Robust-U1’s recovered image safe to trust as evidence?
Only when the corruption is plausibly invertible. The paper’s own limits note that recovery becomes hallucination when evidence is destroyed outright, and that a cleaner-looking image can be wrong in a way the language model is less able to doubt. Its MMMB drop of 1.57 points under 100% corruption, versus 6.06 for Robust-R1, is encouraging, but deployment around safety-critical details still needs uncertainty checks.
One line: MM-OCEAN tells you the model answered for the wrong reasons; Robust-U1 gives it a second chance to look. Read the MM-OCEAN paper and the Robust-U1 paper on arXiv.