Paper Walkthrough: The Personalization Mirage — LLMs Invent a Version of You, and Their Self-Reports Point the Wrong Way

How much of what a memory-equipped LLM 'knows' about you was never actually said? MirageBench judges 143,616 claims across 150 personas and 6 tasks, finding that all 12 tested models over-infer 35–49% of their claims — and that the models reporting the least over-inference tend to fabricate the most.

The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads

From three facts to "your apartment"

Imagine telling a new acquaintance exactly three things about yourself: you're a software engineer, you went rock climbing last weekend, and your cat knocked over your coffee this morning. A few days later, you hear that this acquaintance has been telling people you live in a modern minimalist apartment, prefer nature trips over city tours, are probably single, and enjoy indie rock. You said none of that.

This vignette, which opens the paper (§1), is a description of what personalized LLMs do routinely. Systems that carry user information across sessions — ChatGPT's memory feature, frameworks like Mem0 and MemGPT — are already mainstream, and the paper argues they all rest on one assumption: that a model can reliably tell apart what it knows about a user from what it is guessing. What this study shows, with measurements, is that the assumption does not hold.

What is over-inference?

The paper's central concept is over-inference (OI): generating claims about a user that go beyond what the available evidence supports (§1). It is distinct from classic hallucination (false facts about the world) and from social bias (group-level stereotypes). Over-inference sits in the middle ground between them: fabricating individual-level attributes that feel personalized but were never grounded in anything the user said.

To make this measurable, the paper decomposes model output into individual claims and classifies each one by its relationship to the evidence (§3):

• Grounded: restates what the user said. "The user enjoys rock climbing" ← "I went rock climbing last weekend and it was amazing!"
• Reasonable: one common-sense inference step. "Might be a night owl" ← "Been binge-watching this anime all night, it's 4am"
• Stereotype: fills the gap with occupational or demographic priors instead of individual evidence. "As a software engineer, your apartment probably has a modern, functional design"
• Fabricated: no evidential basis at all. "You're looking to meet someone who shares your passion for adventure" ← the user only mentioned climbing and a cat

The bottom two categories (Stereotype + Fabricated) jointly constitute over-inference, and the primary metric is their share:

Stated in words: of everything model $M$ — whichever system is being scored — said about this person, what share was either filled in from a stereotype or made up outright? The denominator is every claim the model produced about that user; the numerator counts only the two unsupported buckets, Stereotype and Fabricated. A single division, nothing more.

If you've studied machine learning, this shape should feel familiar. Fit an expressive model to only three data points and it will happily "learn" structure the data never supported — overfitting, playing out at the level of a user profile.

MirageBench: the measurement apparatus

Measuring this requires evidence sparse enough that unwarranted personalization becomes visible, tasks that are realistic, and a verdict that doesn't depend on the models being tested. The paper's benchmark, MirageBench, is built accordingly (§4):

• 150 personas. Each pairs a ground-truth profile $P$ of 15 attributes with exactly three revealed facts $E$. Fixing it at three is deliberate: it recreates the early-interaction regime where the model knows an occupation and a hobby or two, but 12 attributes remain unmentioned — exactly where the temptation to personalize is strongest and least reliable (§3). The 150 personas are balanced into three groups of 50: stereotypical (a female nurse who enjoys yoga), counter-stereotypical (a male kindergarten teacher who competes in powerlifting), and neutral (§4.1)
• Six personalization tasks. Writing a dating profile, recommending a weekend itinerary, drafting a recommendation letter, choosing a 100-dollar birthday gift, describing the user's apartment, and naming their biggest stress source. Together they span an "imagination gradient" — from answerable-from-evidence to pure conjecture (§4.2)
• An independent judge. Every claim from every model is classified by a model outside the leaderboard (Claude-Opus-4-7, temperature 0) using the same taxonomy. The judge was validated against a blind human annotator on 400 stratified claims: Cohen's κ = 0.863 on the four-way labels, κ = 0.900 on the binary over-inference decision (§4.3). On this foundation, 143,616 claims from 12 models across 7 families were scored

The pipeline has three stages plus one extension: Probe asks models explicitly what they can infer; Task has them perform the six tasks and then audit their own output; Judge supplies external adjudication; and Accum tracks memory growth over eight rounds of continued conversation (§4.3).

Finding 1: no model escapes

The leaderboard's central result (§5.1, Table 1) is blunt: every one of the 12 models over-infers 35–49% of its personalized claims, with a cross-model mean of 41.6% (41.8% claim-weighted). Even the best performers — Gemini-3.1-pro at 35.1% and Claude-Opus-4-6 at 35.4% — exceed one third; Qwen3-8B tops the table at 48.7%.

The composition cuts deeper. Only 24–31% of claims are grounded in evidence. From the user's perspective, roughly three-quarters of what a personalized model "knows" about them was never communicated. And fabrication (mean 31.1%) dominates stereotyping (mean 10.5%): models invent more than they pigeonhole. The persona design matters too — stereotypical personas draw 44.8% OI versus 37.0% for counter-stereotypical ones, a gap present in all 12 models at +4.0 to +10.5 points (§5.1, Appendix E). The more a person matches the statistical mold, the more freely the mold is used to fill them in.

Finding 2: the Self-Monitoring Inversion

An obvious defense suggests itself: have the model audit its own claims before writing them to memory. The paper tests this head-on, asking each model to label its own output with the same taxonomy and comparing self-reported OI against judge-measured OI (§5.2, Table 2).

The result is the phenomenon in the paper's title. At the model-selection level, self-reported OI is negatively rank-correlated with judge-measured OI — Spearman ρ = −0.60 (p = 0.044; the paper itself labels this exploratory, with a wide bootstrap CI of [−0.90, +0.06] and n = 12). The concrete cases are striking: Qwen3-8B reports the lowest OI of all (13.0%) yet receives the worst judge score (48.7%), while Kimi-K2.5 reports the highest (58.2%) but lands mid-pack under the judge (43.1%).

The proposed mechanism is differential self-labeling strictness. "Strict" self-auditors like Claude, GLM, and Kimi readily flag their own inferences as problematic — and that same metacognitive caution plausibly restrains their generation. "Lenient" auditors like GPT-4o-mini and Qwen3-8B label nearly everything "reasonable" and generate accordingly. Self-reported magnitudes thus reflect labeling calibration, not behavior — which makes them useless, even inverted, as a cross-model safety comparison (§6.2).

This is not a blanket dismissal, though. Within a single model, self-audit still ranks that model's own claims moderately to strongly (AUROC 0.58–0.83, with 9 of 12 models above 0.75; §5.2). And under Probe — when models are asked directly "what can you infer from these 3 facts, with evidence links?" — error rates fall to just 0.7–4.6%, a mean 38.6-point gap versus task-time behavior. The models know; they just don't apply it unless asked. The paper calls this behavioral dishonesty (§6.4).

Finding 3: the more imagination a task demands, the more gets invented

Across the six tasks, OI rates run from 27% to 59% along a clean gradient (§5.3, Table 3). Gift recommendation, which can anchor on stated hobbies, sits at 27.0% ("since you mentioned rock climbing, a chalk bag would make a great gift"). Describing the apartment reaches 57.8% — the model must conjure physical details of a space it knows nothing about, and this task also has the highest stereotype share (19.8%), because occupational clichés are the only "signal" available. The recommendation letter (48.2% OI) is a different pathology: its fabrication is driven not by stereotypes but by genre conventions. A convincing letter must cite specific achievements — "demonstrated exceptional leadership," "mentored junior colleagues" — none of which can be grounded in three casual facts about hobbies and a pet. The format itself demands invention.

Finding 4: inferences accumulate, and are almost never revised

The Accum pilot follows memory over 8 conversation rounds (§5.4). Starting from three facts, GPT-5.5 and GLM-5.1 build profiles of more than 120 inferred attributes by round 8. Growth is close to linear for 9 of 12 models ($R^{2}>0.90$, slopes of 5–15 attributes per round). And the fastest accumulators barely revise: removal rates for the top five are 0.4–5%, and no inference was ever observed being retracted because a later turn contradicted it — GPT-5.5 kept "urban professional lifestyle" from round 2 even after the persona said in round 7 they were visiting a big city "for the first time" (Appendix F). Two important caveats the paper states repeatedly: the pilot uses only 2 personas, and the memory prompt itself instructs models not to remove entries unless directly contradicted (Appendix G.7), which biases the setup toward accumulation. Read it as suggestive, not definitive.

Why do models over-infer?

The paper offers three mechanisms, often operating together (§6.1). First, the verbosity trap: longer responses mechanically produce more claims from the same three facts, and output length correlates with OI (r = 0.59 in self-audit data) — but even the most concise model still hits 45.1%, so length is not the cause; the problem is that what gets said is unanchored. Second, pretraining priors as gap-fillers: with sparse evidence, models reach for distributional knowledge (an engineer "probably" has a minimalist apartment) — and applying group statistics to an individual is precisely what stereotyping is. Third, genre expectations demand fabrication: for recommendation letters or dating profiles, refusing to elaborate would be unhelpful, so the format creates an obligation to invent — consistent, the paper notes, with findings that RLHF amplifies compliance with user and genre expectations.

Limitations

The paper's own limitations section (Appendix A) deserves attention. (1) There is a single judge model and a single human validator; inter-annotator agreement is unmeasured. (2) Scoring treats only the three revealed facts as evidence, so task-presupposed claims ("planning a weekend trip" in the itinerary task) count as over-inference — OI rates are an upper bound relative to a task-aware notion of grounding. (3) The Self-Monitoring Inversion is exploratory: n = 12 with correlated model families, and the bootstrap CI includes zero. (4) Accum is a 2-persona pilot with a retention-biased prompt. (5) No mitigations are evaluated — the paper characterizes the phenomenon and stops there.

How this is used in practice

The people this paper speaks to most directly are engineers building memory-equipped assistants or personalization features. Translating its implications (§6.3) into working terms:

• Tag memory writes with provenance. The paper proposes three epistemic statuses — stated (user said it), inferred (with an evidence link), generated (no evidence) — and argues that unlinked inferences should be treated as hypotheses, not facts. That much is the paper's claim; in practice, if you run your own memory layer on top of Mem0- or MemGPT-style storage, adding a source_turn_id field to each record is enough to start
• Never pick a model by its self-reported confidence. "Our model self-assesses as making very few inferences" is, on this paper's measurements, a signal pointing the wrong way. Compare models with an external judge, always
• Do use within-model self-audit. Filtering a model's own claims by its self-audit score before a memory write is supported (AUROC 0.58–0.83) — but thresholds must be calibrated per model. Reusing one threshold across models is the classic mistake this paper warns against
• If you use LLM-as-judge, pair it with human validation. The κ = 0.863 figure exists because the authors ran a stratified 400-claim blind re-annotation. The same procedure — stratified sample, blind pass, compute κ — is the reproducible way to certify your own judging pipeline
• Design tasks to lower fabrication pressure. If your product generates imagination-heavy formats (bios, recommendation text), constraining the prompt to "mention only revealed facts" is worth trying — though note this is a practical extrapolation from the 27%→59% gradient, not a mitigation the paper tested

Related articles

• Overfitting and Evaluation — the source of this article's central analogy: models inventing structure the data doesn't support
• LLM Agents Basics — the anatomy of agents with memory, the setting this paper stress-tests
• Instruction Tuning and RLHF — the training framework implicated in amplifying compliance with genre expectations