Paper Walkthrough: The Personalization Mirage — LLMs Invent a Version of You, and Their Self-Reports Point the Wrong Way
How much of what a memory-equipped LLM 'knows' about you was never actually said? MirageBench judges 143,616 claims across 150 personas and 6 tasks, finding that all 12 tested models over-infer 35–49% of their claims — and that the models reporting the least over-inference tend to fabricate the most.
The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
Primary source — what this article is built on
undefined2026-08-05→undefined2026-08-12same month
The Personalization Mirage: How LLMs Fabricate User ProfilesYushi Sun, Yanjie Zhang, Rui Sheng · 2026-08-05 · v1"arXiv:2608.04570Paper page·PDFhttps://arxiv.org/abs/2608.04570"and Why Self-Monitoring Misleads
undefined
Personalized LLMs with persistent memory are increasingly deployed, yet the faithfulness of their user models remains unexamined. We study over-inference (OI): the phenomenon where LLMs fabricate user attributes beyond what evidence supports. We introduce MirageBench, comprising 150 personas balanced across stereotypical, counter-stereotypical, and neutral profiles, 6 personalization tasks spanning an ``imagination gradient'', a four-way faithfulness taxonomy operationalized by an independent judge (validated against a blind human annotator on 400 claims: Cohen's kappa = 0.863 four-class, kappa = 0.900 binary), and a leaderboard of 12 models across 7 families on 143616 judged claims. We find that over-inference is pervasive: every one of the 12 models over-infers 35%--49% of its claims (cross-model mean 41.6%; claim-weighted 41.8%), with no model in this evaluation escaping it. Most strikingly, we surface a Self-Monitoring Inversion: at the model-selection level, models' self-assessed OI is negatively rank-correlated with their judge-measured OI (rho = -0.60, p = 0.044; exploratory, wide bootstrap CI [-0.90, +0.06], n = 12). The models that report the least over-inference tend to be flagged as fabricating the most, so self-reported confidence is a misleading signal for comparing models, even though within a single model self-audit still ranks that model's own claims moderately well (AUROC 0.58--0.83). We further show that OI is task-dependent (27%--59%) and that, in a multi-turn pilot, inferred attributes accumulate approximately linearly with little revision. MirageBench positions external verification, rather than model self-report, as a more reliable foundation for trustworthy personalization.
From three facts to "your apartment"
Imagine telling a new acquaintance exactly three things about yourself: you're a software engineer, you went rock climbing last weekend, and your cat knocked over your coffee this morning. A few days later, you hear that this acquaintance has been telling people you live in a modern minimalist apartment, prefer nature trips over city tours, are probably single, and enjoy indie rock. You said none of that.
This vignette, which opens the paper (§1), is a description of what personalized LLMs do routinely. Systems that carry user information across sessions — ChatGPT's memory feature, frameworks like Mem0 and MemGPT — are already mainstream, and the paper argues they all rest on one assumption: that a model can reliably tell apart what it knows about a user from what it is guessing. What this study shows, with measurements, is that the assumption does not hold.
What is over-inference?
The paper's central concept is over-inference (OI): generating claims about a user that go beyond what the available evidence supports (§1). It is distinct from classic hallucination (false facts about the world) and from social bias (group-level stereotypes). Over-inference sits in the middle ground between them: fabricating individual-level attributes that feel personalized but were never grounded in anything the user said.
To make this measurable, the paper decomposes model output into individual claims and classifies each one by its relationship to the evidence (§3):
- Grounded: restates what the user said. "The user enjoys rock climbing" ← "I went rock climbing last weekend and it was amazing!"
- Reasonable: one common-sense inference step. "Might be a night owl" ← "Been binge-watching this anime all night, it's 4am"
- Stereotype: fills the gap with occupational or demographic priors instead of individual evidence. "As a software engineer, your apartment probably has a modern, functional design"
- Fabricated: no evidential basis at all. "You're looking to meet someone who shares your passion for adventure" ← the user only mentioned climbing and a cat
The bottom two categories (Stereotype + Fabricated) jointly constitute over-inference, and the primary metric is their share:
Stated in words: of everything model — whichever system is being scored — said about this person, what share was either filled in from a stereotype or made up outright? The denominator is every claim the model produced about that user; the numerator counts only the two unsupported buckets, Stereotype and Fabricated. A single division, nothing more.
If you've studied machine learning, this shape should feel familiar. Fit an expressive model to only three data points and it will happily "learn" structure the data never supported — overfitting, playing out at the level of a user profile.
MirageBench: the measurement apparatus
Measuring this requires evidence sparse enough that unwarranted personalization becomes visible, tasks that are realistic, and a verdict that doesn't depend on the models being tested. The paper's benchmark, MirageBench, is built accordingly (§4):
- 150 personas. Each pairs a ground-truth profile of 15 attributes with exactly three revealed facts . Fixing it at three is deliberate: it recreates the early-interaction regime where the model knows an occupation and a hobby or two, but 12 attributes remain unmentioned — exactly where the temptation to personalize is strongest and least reliable (§3). The 150 personas are balanced into three groups of 50: stereotypical (a female nurse who enjoys yoga), counter-stereotypical (a male kindergarten teacher who competes in powerlifting), and neutral (§4.1)
- Six personalization tasks. Writing a dating profile, recommending a weekend itinerary, drafting a recommendation letter, choosing a 100-dollar birthday gift, describing the user's apartment, and naming their biggest stress source. Together they span an "imagination gradient" — from answerable-from-evidence to pure conjecture (§4.2)
- An independent judge. Every claim from every model is classified by a model outside the leaderboard (Claude-Opus-4-7, temperature 0) using the same taxonomy. The judge was validated against a blind human annotator on 400 stratified claims: Cohen's κ = 0.863 on the four-way labels, κ = 0.900 on the binary over-inference decision (§4.3). On this foundation, 143,616 claims from 12 models across 7 families were scored
The pipeline has three stages plus one extension: Probe asks models explicitly what they can infer; Task has them perform the six tasks and then audit their own output; Judge supplies external adjudication; and Accum tracks memory growth over eight rounds of continued conversation (§4.3).
Comments
Sign in to comment