Alignment · Language Models · Multimodal Models

Personality Questionnaires vs Grounded Evaluation for LLMs, With Numbers

Two papers test whether human-style personality instruments work on AI: BFI/PVQ self-reports agree with real LLM behavior at only Spearman 0.26-0.31, and 51.3% of MLLMs' correct personality ratings rest on wrong cues.

Personality Questionnaires vs Grounded Evaluation for LLMs, With Numbers

Two instruments that promise the same thing

A strange industry has grown up around the idea that you can profile a language model the way you profile a person. Feed an LLM the Big Five Inventory or the Portrait Values Questionnaire, read off its Likert self-ratings, and publish its “personality.” Point a multimodal model at a short video of a human and ask it to rate extraversion, then use that score in hiring screens or ad targeting. Both practices share one assumption: that a tidy, human-validated instrument keeps its meaning when the subject is a model.

The two papers behind this guide set out to check that assumption, and they check it in different media with different instruments. Psychometric questionnaires mischaracterize LLM behavior (Seoul National University, 2025) compares questionnaire self-reports against behavior measured from generation probabilities on realistic prompts, across eight open models. Perception or Prejudice (2026) builds MM-OCEAN, a 1,104-video benchmark that scores whether an MLLM’s Big Five ratings are actually anchored in observable behavior, across 27 models. Neither paper argues that models have no dispositions. Both argue that the instruments we use to measure those dispositions quietly stopped measuring behavior.

Key numbers

AxisPsychometric questionnaires (BFI / PVQ)MM-OCEAN grounded reasoning
InstrumentValidated human self-report surveys: 44- and 10-item BFI, 40- and 21-item PVQ1,104 videos, 5,320 multiple-choice grounding questions, about 13,500 human-verified behavioral observations
Claimed targetStable values and Big Five personality dispositionsApparent-personality ratings grounded in observable behavior
Headline failureSelf-report vs real-query behavior agree at Spearman 0.26 (BFI) and 0.31 (PVQ)51.3% mean Prejudice Rate; 10.4% mean Holistic-Grounding Rate
Within-method baselineTwo versions of the same survey agree at 0.74-0.77Human inter-annotator agreement on verdict labels is 77%
Root cause shownItems leak lexical cues: construct-recognition F1 is 0.69-0.83 on survey items but 0.09 on realistic scenariosPerception is the bottleneck: spatial localization 30.7%, micro-expression detection 34.6%, temporal-causal reasoning 64.8%
Factor structureItem consistency eta-squared 0.526 on questionnaires collapses to 0.040 on real-query generation (p < 0.01)Four separate failure rates (PR, CR, IR, HR) instead of one accuracy score
Sensitivity checkPersona steering moves questionnaire answers 62/80 (p < 0.001) but real generation 40/80 (p = 0.54, chance)Closed-source beats open by 5.6 points on rating but 26.6 points on grounding
Best resultNo questionnaire profile predicted behavior above 0.31Gemini 3 Flash leads at 33.5% HR with 17.2% PR; best open model (Qwen3.5-397B) reaches 15.9% HR with a 41.5% PR

Read the middle column as the questionnaire paper’s core claim: the method you pick moves the answer more than the trait you measure. The 0.74-0.77 agreement between two versions of the same survey is what a valid instrument looks like; 0.26-0.31 is what the questionnaires actually deliver against behavior. Read the right column as the multimodal mirror image: rating accuracy and genuine perception are almost decoupled, which is exactly what a single leaderboard score would have hidden.

Why questionnaires produce tidy, hollow numbers

The mechanism in the questionnaire paper is worth understanding because it generalizes far beyond personality tests. Human survey items are deliberately transparent about what they probe: “It is important to him to be loyal” announces the value being measured. That transparency is a feature for human respondents, who cannot game a questionnaire they have memorized the scoring key for. A language model has read everything, including the psychometrics literature. Give it a BFI item and it recognizes the construct at F1 0.69-0.83, then emits the socially desirable answer. Strip the cue away, as a genuine user query does, and the same construct recognition drops to 0.09 while the apparent trait stops predicting anything.

The persona experiment is the cleanest demonstration. Steer a model with a demographic persona and its questionnaire answers shift in human-like directions on 62 of 80 checks (p < 0.001). Measure what the steered model actually generates on realistic prompts and the same personas move behavior on only 40 of 80 checks (p = 0.54, indistinguishable from chance). The questionnaire sees a personality that the behavior does not have. The paper’s own numbers, not interpretation, carry the point: internally tidy (eta-squared 0.526), behaviorally hollow (0.040).

Why MLLMs get ratings right for the wrong reason

The MM-OCEAN paper lands the same punch in the visual domain. Its Grounded Personality Reasoning task forces a model to do three things at once: produce a 1-5 Big Five rating (T1), explain it by citing observable behavior (T2), and answer structured questions pinning the judgment to concrete cues like micro-expressions, spatial location, and temporal-causal chains (T3). The headline metric is the Prejudice Rate: the fraction of correct ratings that are anchored in the wrong cues. Across 27 models it averages 51.3%. Over half of all correct personality ratings are, in the paper’s framing, lucky guesses from first impressions.

The decomposition is the contribution. A single accuracy number would have shown models scoring plausibly on personality; splitting the output into PR, Confabulation Rate, Integration-failure Rate, and Holistic-grounding Rate shows where they break. The bottleneck is perception, not reasoning: temporal-causal reasoning reaches 64.8% mean accuracy while spatial localization sits at 30.7% and micro-expression detection at 34.6%. Models reason about behavior far better than they perceive it. And the closed-vs-open gap (5.6 points on rating, 26.6 points on grounding) confirms that the expensive capability is locating evidence, not emitting a plausible score. Even the leader, Gemini 3 Flash at 33.5% HR, fails to fully ground its judgment two times out of three.

The shared lesson: construct validity is not behavioral validity

Put side by side, the two papers are the same experiment run on different instruments. The questionnaire paper shows a measure can be construct-valid in the human sense, meaning consistent, factorable, and persona-responsive, and still fail to predict what the model does. The benchmark paper shows an output can be accurate in the leaderboard sense and still be produced without perception. Both failure modes feel like measurement successes right up until you check them against behavior or evidence.

This is why both papers resist the comforting fallback of “models have no personality anyway.” The questionnaire authors note their result shows questionnaires fail to predict behavior, not that no stable dispositions exist; a genuinely behavioral instrument might recover something real. The MM-OCEAN authors note their benchmark measures apparent personality in 15-second single-speaker English clips, not ground truth, and is diagnostic rather than prescriptive. What both rule out is narrower and more useful: a tidy score is not evidence of the thing it claims to measure, and anyone using such a score for certification, hiring, or targeting is measuring the instrument’s surface, not the model’s conduct.

When to use which

Use neither instrument to certify anything. The defensible uses are narrower:

  • Use BFI/PVQ-style questionnaires only as a red-team probe of how a model handles leading, cue-laden wording. That is what they actually measure: cue recognition. If you need a values or behavior audit, use behavioral probes on realistic prompts without lexical giveaways, which is the harder but more trustworthy bar the first paper proposes.
  • Use MM-OCEAN-style grounding decomposition when you evaluate any multimodal system whose outputs justify decisions about people. Report the Holistic-grounding Rate and Prejudice Rate, not the rating accuracy. A 5.6-point rating gap between closed and open models is noise next to a 26.6-point grounding gap.
  • Use single leaderboard scores for neither. Both papers show that aggregation is exactly where the failure hides: within-survey agreement looks fine at 0.74-0.77 while behavior agreement is 0.26-0.31; rating accuracy looks fine while 51.3% of correct ratings rest on prejudice.

Limits and open questions

The questionnaire study covers eight open models from four families (Gemma 3, Qwen 2.5, Qwen 3, GPT-OSS); GPT-4-class, Claude, and Gemini are absent, and frontier closed models may behave differently under either probe. Its behavioral measure, generation probability over value-laden options, is itself a constructed proxy rather than raw deployment behavior. The everyday-query scenarios were curated by the authors, so coverage and cultural framing are open.

MM-OCEAN measures apparent personality in short, monolingual, single-speaker clips from one dataset, not self-reported ground-truth traits; a model could be “wrong” in ways humans are also wrong. Its T2 reasoning quality is judged by GPT-4o-mini rather than humans, importing that judge’s blind spots. And it is diagnostic, not prescriptive: it shows the prejudice gap clearly but proposes no training method to close it, leaving whether grounding can be trained in as the open question. Together the two papers also leave the constructive question wide open: if you wanted a genuinely behavioral personality or values instrument for models, nobody has built one yet.

FAQ

Do LLMs have personalities? A comparison of two evaluation methods

These two papers do not answer that. They show that the instruments commonly used to assign personalities (Big Five questionnaires, first-impression video ratings) produce scores that fail to track behavior or evidence. Whether models have stable dispositions that a better instrument could recover remains open.

Can I trust a Big Five score? Questionnaire vs behavior results

Not as a behavior predictor. In the questionnaire study, BFI self-report profiles agreed with real-query behavior at Spearman 0.26, versus 0.77 between two BFI versions. The score is internally consistent but does not track what the model does, because items carry lexical cues the model recognizes (F1 0.69-0.83) and pattern-matches.

What is the Prejudice Rate in MM-OCEAN and how does the benchmark work?

The fraction of correct personality ratings that are not anchored in the right behavioral cues. Across 27 models it averages 51.3%, meaning most correct ratings are right answers arrived at for the wrong reasons. The best model, Gemini 3 Flash, still has a 17.2% Prejudice Rate.

Why is grounding harder than rating? The method-level difference

Rating only needs a plausible score; grounding requires locating specific visual evidence. The closed-vs-open gap is 5.6 points on rating but 26.6 points on grounding, and the weakest perceptual categories are spatial localization (30.7%) and micro-expression detection (34.6%), far below temporal-causal reasoning (64.8%).

Does persona prompting change model behavior or just benchmark answers?

Mostly it changes how the model answers questionnaires. Persona steering shifted questionnaire responses in human-like directions on 62 of 80 checks (p < 0.001) but moved actual generation behavior on only 40 of 80 (p = 0.54, chance). Paper personas are real; behavioral personas, on this evidence, are largely cosmetic.