Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox

METAL MEDIA

PersonalBench: Measuring the Authorship Gap in LLM Personalization

arXiv:2608.197462026-08-21

AI can be prompted to write 'like someone,' but its own voice never fully disappears

Researchers tested several ways of prompting large language models to write in a specific person's style and measured the results with an authorship-detection AI, an LLM judge, and classic writing-style statistics. The generated texts could be told apart by target person, but compared to that person's real writing they still looked as different as two random strangers, or worse. The team released PersonalBench, a benchmark built on 50 authors and 1,000 generated texts, showing that prompt-based personalization tweaks style within the model's own voice but never actually crosses into sounding human-authentic.

METAL MEDIA explanatory visual

AI can be prompted to write 'like someone,' but its own voice never fully disappears

  1. 01Problem: existing personalization benchmarks measure whether an LLM completes a task well or matches stated preferences, not whether the output actually reads like the target person wrote it
  2. 02Method: combined an authorship-verification model called LUAR, an LLM acting as judge, and function-word/punctuation statistics to score four personalization techniques (no personalization, few-shot examples, style-profile extraction, contrastive examples) across Qwen 3 and GLM-4 models, 50 authors, 1,000 generations
  3. 03Result: generated texts were distinguishable by target author among themselves (AUC 0.918), but similarity to the real author's actual writing (0.484-0.508) fell below even the baseline similarity between two unrelated real humans (0.626)
  4. 04Twist: the LLM judge rated the profile-extraction method as the clear winner, but this turned out to be circular - the judge and the method used the same trait-extraction process - and the authorship-detection model found no real advantage
  5. 05Limits: both tested models were 32-billion-parameter, compressed (quantized) versions; the writing samples came only from early-2000s blogs; the LLM judge itself hasn't been checked against human raters
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

  1. Problem: existing personalization benchmarks measure whether an LLM completes a task well or matches stated preferences, not whether the output actually reads like the target person wrote it
  2. Method: combined an authorship-verification model called LUAR, an LLM acting as judge, and function-word/punctuation statistics to score four personalization techniques (no personalization, few-shot examples, style-profile extraction, contrastive examples) across Qwen 3 and GLM-4 models, 50 authors, 1,000 generations
  3. Result: generated texts were distinguishable by target author among themselves (AUC 0.918), but similarity to the real author's actual writing (0.484-0.508) fell below even the baseline similarity between two unrelated real humans (0.626)
  4. Twist: the LLM judge rated the profile-extraction method as the clear winner, but this turned out to be circular - the judge and the method used the same trait-extraction process - and the authorship-detection model found no real advantage
  5. Limits: both tested models were 32-billion-parameter, compressed (quantized) versions; the writing samples came only from early-2000s blogs; the LLM judge itself hasn't been checked against human raters
Figure 1: LUAR authorship similarity by method (5-post aggregation). All methods score below the real-text cross-author floor (0.626), far from the real-author ceiling (0.756). The total spread across methods is only 0.024. The LLM’s authorship fingerprint is so dominant that generated text sits in a distinct regime below human-to-human similarity.
Figure 1: LUAR authorship similarity by method (5-post aggregation). All methods score below the real-text cross-author floor (0.626), far from the real-author ceiling (0.756). The total spread across methods is only 0.024. The LLM’s authorship fingerprint is so dominant that generated text sits in a distinct regime below human-to-human similarity.
Table 1: Effect of prompt construction on baseline scores. Naïve extraction leaks author voice, inflating the unpersonalized baseline by 28pp on same-author judgments.
Prompt typeNon-personalizedFew-shot vs. baseline
TMRSA%ΔTMRWin%
Raw first sentence0.58750%−0.08023%
Content summary0.38422%+0.04933%
Figure 2: Distribution of per-generation LUAR similarity scores by method. All four methods produce overlapping distributions centered near 0.44 (1v5 per-generation scores), well below the real-author distribution centered at 0.58.
Figure 2: Distribution of per-generation LUAR similarity scores by method. All four methods produce overlapping distributions centered near 0.44 (1v5 per-generation scores), well below the real-author distribution centered at 0.58.
Table 2: Method comparison across 50 authors, 1,000 generations. LUAR similarity uses 5-post aggregation (5 generations per author-method pair compared to 5 training posts) and is the primary authorship metric (↑ better). TMR = trait match rate from LLM judge. SA% = same-author rate. FuncCos = function word cosine similarity. Ceiling = real author’s test posts; floor = cross-author random pairs. All methods score below the real-text cross-author floor on LUAR despite appearing differentiated on TMR. CIs use hierarchical bootstrap (resampling authors, then generations within authors) to account for within-author correlation (B=10,000).
MethodLUAR ↑TMR ↑SA% ↑FuncCos ↑
Non-Personalized0.484±.0190.384±.05822%±70.741±.011
Few-Shot0.508±.0200.433±.06131%±80.749±.011
Profile Extraction0.502±.0190.542±.06029%±80.761±.010
Contrastive0.494±.0200.447±.05936%±80.752±.011
Real Author (ceiling)0.7560.42730%—
Cross-Author (floor)0.6260.3907%—
Figure 3: Calibration across three metrics. Each panel shows method scores (bars) with ceiling (real author, green) and floor (cross-author, red) baselines. On LUAR (left), all methods score below the real-text cross-author floor. On TMR (center), profile extraction exceeds the real-author ceiling—evidence of circularity (§4.6.1). On SA% (right), contrastive leads but all methods exceed the chance floor.
Figure 3: Calibration across three metrics. Each panel shows method scores (bars) with ceiling (real author, green) and floor (cross-author, red) baselines. On LUAR (left), all methods score below the real-text cross-author floor. On TMR (center), profile extraction exceeds the real-author ceiling—evidence of circularity (§4.6.1). On SA% (right), contrastive leads but all methods exceed the chance floor.
Table 3: Pearson correlation between evaluation metrics (n=1,000). All pairwise correlations are near zero, indicating the metrics capture fundamentally different constructs. No single metric is a reliable proxy for another.
LUARTMRFuncCos
LUAR1.00——
TMR0.0131.00—
FuncCos0.0260.0671.00
Figure 4: LUAR similarity vs. TMR for all 1,000 generations, colored by method. No systematic relationship (r=0.013). Profile extraction’s high TMR does not correspond to high LUAR.
Figure 4: LUAR similarity vs. TMR for all 1,000 generations, colored by method. No systematic relationship (r=0.013). Profile extraction’s high TMR does not correspond to high LUAR.

Why it matters

This matters for anyone building writing assistants, chatbots, or brand-voice tools that claim to adapt to a specific person's style, because it suggests clever prompting alone may not be enough to genuinely capture someone's voice. It also gives the field a calibrated way to measure whether a personalization method actually works instead of just looking convincing.

Terms in this paper

  • LUAR · an AI model trained on millions of Reddit posts to judge whether two texts were written by the same person
  • LLM-as-judge · using one language model to evaluate another model's output instead of a human evaluator
  • AUC · a score measuring how well a method distinguishes between two groups, where higher (closer to 1) means better separation
  • inference-time personalization · adjusting a model's output through prompts and examples at generation time, without retraining the model's weights
  • stylometrics · statistical analysis of writing habits like word choice frequency and punctuation patterns used to identify authorship

Original abstract (English)

Personalized text generation aims to make LLMs write in a specific individual's style, yet existing benchmarks measure task accuracy or preference alignment rather than whether the model's output actually resembles the target author's writing. We introduce PersonalBench, a benchmark that evaluates inference-time personalization methods through three independent lenses: LUAR (a trained authorship verification model), an LLM-as-judge, and automated stylometrics. Across 50 authors, 1,000 generations, and two model families (Qwen 3, GLM-4), we find that personalization methods do produce author-differentiated output (LUAR discriminates target authors within generated text at AUC=0.918) but this differentiation never crosses the human-LLM boundary. All methods achieve LUAR similarity to real authors in the range 0.484-0.508, below the cross-author human floor of 0.626 (ceiling 0.756). The LLM's own authorship fingerprint dominates: generated text is more distant from any human author than random humans are from each other. Methods are statistically indistinguishable from each other on LUAR (spread 0.024) despite appearing differentiated on the LLM judge, a discrepancy we trace to circularity between trait extraction and profile extraction. We validate that LUAR reliably measures authorship in our corpus (AUC=0.76 single-post, 0.96 multi-post). We release PersonalBench as a calibrated measuring stick: inference-time personalization modulates the LLM's style but does not bridge the gap to human authorship.

Authors · Yash Ganpat Sawant

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Yash Ganpat Sawant et al., arXiv:2608.19746, CC BY 4.0