Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox

METAL MEDIA

Generating Diverse Personas for User Simulators to Test Interview Dialogue Systems

arXiv:2608.195492026-08-21

To test interview-style chatbots you need many different fake users, so this work has an LLM automatically generate those fake user personalities

Interview dialogue systems that ask users about things like travel plans or dessert preferences are costly to test with real humans. This paper has an LLM generate a large, diverse set of fake user personas from just a handful of hand-written examples, then uses those personas to drive simulated users. Experiments show the resulting simulated conversations are more varied than with a fixed set of manually written personas.

METAL MEDIA explanatory visual

To test interview-style chatbots you need many different fake users, so this work has an LLM automatically generate those fake user personalities

  1. 01The method was tested on two Japanese interview dialogue systems: one interviewing users about travel, one about sweets preferences.
  2. 02Starting from 10 hand-written 'seed' personas, GPT-4o generated 100 new personas per condition using in-context few-shot prompting.
  3. 03Persona generation also specified two personality traits tied to communication style: degree of anthropomorphism (treating the system as an object vs. as a human) and degree of elaborateness (redundant vs. direct speech).
  4. 04Generated personas drove a GPT-4o-based user simulator talking to a GPT-4o-mini-based interview system, and diversity was measured with metrics like utterance length variation and lexical type-token ratio.
  5. 05Simply generating new personas with an LLM already increased content diversity (content-word type-token ratio rose from .106 to .122 for travel and .109 to .133 for sweets versus the baseline), and adding the elaborateness trait increased stylistic diversity, with utterance length standard deviation rising from 7.0 to 18.2 for travel and 8.0 to 17.7 for sweets.
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

  1. The method was tested on two Japanese interview dialogue systems: one interviewing users about travel, one about sweets preferences.
  2. Starting from 10 hand-written 'seed' personas, GPT-4o generated 100 new personas per condition using in-context few-shot prompting.
  3. Persona generation also specified two personality traits tied to communication style: degree of anthropomorphism (treating the system as an object vs. as a human) and degree of elaborateness (redundant vs. direct speech).
  4. Generated personas drove a GPT-4o-based user simulator talking to a GPT-4o-mini-based interview system, and diversity was measured with metrics like utterance length variation and lexical type-token ratio.
  5. Simply generating new personas with an LLM already increased content diversity (content-word type-token ratio rose from .106 to .122 for travel and .109 to .133 for sweets versus the baseline), and adding the elaborateness trait increased stylistic diversity, with utterance length standard deviation rising from 7.0 to 18.2 for travel and 8.0 to 17.7 for sweets.
Figure 1: Overview of the proposed method.
Figure 1: Overview of the proposed method.
Table 1: Diversity metrics of the simulations for the travel interview system. For example, the row where Condition is APM and Personality is All represents metrics calculated from all the dialogues under the APM condition. The subsequent row, where Personality is High, shows metrics calculated from the dialogues under the APM condition using a high degree of anthropomorphism in the personality setting. The row where Condition is APM+EL and Personality is High+High represents metrics calculated from the dialogues with a high degree of anthropomorphism and a high degree of elaborateness. Bold numbers are mentioned in the main text.
ConditionPersonality#DialoguesAve. utterance length (S.D.)Ave. utterancelength (S.D.)Total wordsTotalwordsUnique wordsUniquewordsUnique bigramsUniquebigramsTTR
Ave. utterance
length (S.D.)
Total
words
Unique
words
Unique
bigrams
BL10028.2(7.7)42,26015,74731,620.373
noPT10028.5(7.0)42,78416,36232,730.382
APMAll10028.4(7.2)42,56316,08932,334.378
High5030.1(7.6)22,5478,40617,098.373
Low5026.7(6.3)20,0167,68315,236.384
ELAll10036.0(18.2)53,95118,55639,010.344
High5050.7(13.9)38,00112,17526,883.320
Low5021.3(5.7)15,9506,38112,127.400
APM+ELAll10031.4(13.4)47,11116,85634,839.358
High+High2546.2(12.3)17,3085,62412,309.325
High+Low2523.6(4.8)8,8363,4516,787.391
Low+High2535.2(9.9)13,1844,6519,796.353
Low+Low2520.8(5.9)7,7833,1305,947.402
Table 2: Diversity metrics of the simulations for the sweets interview system.
ConditionPersonality#DialoguesAve. utterance length (S.D.)Ave. utterancelength (S.D.)Total wordsTotalwordsUnique wordsUniquewordsUnique bigramsUniquebigramsTTR
Ave. utterance
length (S.D.)
Total
words
Unique
words
Unique
bigrams
BL10025.9(7.8)25,15211,13820,534.443
noPT10025.6(8.0)25,50911,10520,634.435
APMAll10025.6(8.2)25,68011,12420,862.433
High5026.5(8.2)13,2695,72310,824.431
Low5024.6(8.0)12,4115,40110,038.435
ELAll10033.9(17.7)34,02013,21426,117.388
High5047.5(14.8)23,7558,57817,808.361
Low5020.3(6.1)10,2654,6368,309.452
APM+ELAll10028.1(12.2)28,06711,54922,182.411
High+High2539.4(13.1)9,8573,7437,643.380
High+Low2522.2(6.7)5,5602,4584,533.442
Low+High2530.6(10.5)7,6423,0876,015.404
Low+Low2520.0(5.9)5,0082,2613,991.451

Why it matters

This lets developers stress-test interview dialogue systems against a wide range of user behaviors without recruiting real human testers, cutting labor and cost. Broader persona diversity increases the chance of catching unexpected system failures before deployment.

Terms in this paper

  • persona · a profile of personality, preferences, and speaking style assigned to a simulated user
  • interview dialogue system · a conversational AI that asks users questions to collect information from them
  • user simulator · a simulated conversational partner that stands in for a real human when testing a dialogue system
  • type-token ratio (TTR) · the ratio of unique words to total words used, a measure of vocabulary diversity
  • few-shot in-context learning · giving a language model a few examples in the prompt so it produces new outputs in a similar style

Original abstract (English)

This paper addresses the issue of the significant labor required to test interview dialogue systems. While interview dialogue systems are expected to be useful in various scenarios, like other dialogue systems, testing them with human users requires significant effort and cost. Therefore, testing with user simulators can be beneficial. Since most conventional user simulators have been primarily designed for training task-oriented dialogue systems, little attention has been paid to the personas of the simulated users. During development, testing interview dialogue systems requires simulating a wide range of user behaviors, but manually creating a large number of personas is labor-intensive. We propose a method that automatically generates personas for user simulators using a large language model. Furthermore, by assigning personality traits related to communication styles when generating personas, we aim to increase the diversity of communication styles in the user simulator. Experimental results show that the proposed method enables the user simulator to generate utterances with greater variation.

Authors · Mikio Nakano, Kazunori Komatani, Hironori Takeuchi

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Mikio Nakano et al., arXiv:2608.19549, arxiv-nonexclusive