Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox

METAL MEDIA

Are LLMs becoming similarly creative? Evidence from three years of models

arXiv:2608.194372026-08-21

Newer AI chatbots are giving increasingly similar answers to each other, three years of data show

Researchers tested 68 AI models released since 2023 with creativity-focused prompts and measured how different their answers were from one another. They found a statistically significant trend: answers from different model providers have become more similar over time. If this trend continues, the researchers warn, AI could narrow the range of ideas people encounter when using it for creative work.

METAL MEDIA explanatory visual

Newer AI chatbots are giving increasingly similar answers to each other, three years of data show

  1. 01The team used two prompt sets — the Alternate Uses Task (a classic psychology test asking for unusual uses of everyday objects) and Infinity-Chat100 (100 real user questions) — and ran them through 68 models from 12 providers
  2. 02Each response was converted into a vector using a sentence-embedding model, then compared using cosine distance; release dates were grouped into nine time bins for regression analysis
  3. 03Only models from different companies were compared (to avoid confounds from shared architecture or training data), and results were checked with 1,000 rounds of resampling so no single company's models dominated the trend
  4. 04On the Alternate Uses Task, average distance between different companies' answers fell from about 0.50 in the earliest period to below 0.40 recently; on Infinity-Chat100 it fell more gently from about 0.34 to 0.32, with all 1,000 resampling runs confirming a declining trend
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

  1. The team used two prompt sets — the Alternate Uses Task (a classic psychology test asking for unusual uses of everyday objects) and Infinity-Chat100 (100 real user questions) — and ran them through 68 models from 12 providers
  2. Each response was converted into a vector using a sentence-embedding model, then compared using cosine distance; release dates were grouped into nine time bins for regression analysis
  3. Only models from different companies were compared (to avoid confounds from shared architecture or training data), and results were checked with 1,000 rounds of resampling so no single company's models dominated the trend
  4. On the Alternate Uses Task, average distance between different companies' answers fell from about 0.50 in the earliest period to below 0.40 recently; on Infinity-Chat100 it fell more gently from about 0.34 to 0.32, with all 1,000 resampling runs confirming a declining trend
Figure 1: Methodology overview: we select open-ended prompts designed to elicit creative behavior from models; run these against different model lineages; embed model outputs in a semantic space to compute their semantic distance; and perform a regression analysis to understand changes in semantic distance over time.
Figure 1: Methodology overview: we select open-ended prompts designed to elicit creative behavior from models; run these against different model lineages; embed model outputs in a semantic space to compute their semantic distance; and perform a regression analysis to understand changes in semantic distance over time.
Table 1: Bootstrap OLS estimates for mean cosine distance across nine model release bins.
DatasetModels (Pairs)Slope per Bin95% CI
Alternate Uses Task68 (273)−0.01385[−0.01695,−0.01044]
Infinity-Chat67 (268)−0.00167[−0.00267,−0.00074]
Figure 2: Regression results and bootstrapped slope estimates for AUT and Infinity-Chat response distances across model generations. Our findings of consistently negative slopes across the observation period indicate semantic convergence across LLM creative outputs over time.
Figure 2: Regression results and bootstrapped slope estimates for AUT and Infinity-Chat response distances across model generations. Our findings of consistently negative slopes across the observation period indicate semantic convergence across LLM creative outputs over time.
Table A1: Observed and missing responses for models with at least one missing response. Each model had 10 possible AUT responses and 100 possible Infinity-Chat100 responses.
AUTInfinity-Chat100
ModelObservedMissingObservedMissing
minimax/minimax-010100100
minimax/minimax-m1640100
mistralai/mistral-small-3.1-24b-instruct1008911
qwen/qwen3.6-max-preview281000
qwen/qwen3-max100946
meta-llama/llama-3.2-3b-instruct100973
anthropic/claude-fable-5100991
minimax/minimax-m2.1100991
qwen/qwen-2.5-72b-instruct100991
All 69 models668226,677223
Figure A1: Distribution of model releases over time across the 12 model providers and 68 versions in our dataset.
Figure A1: Distribution of model releases over time across the 12 model providers and 68 versions in our dataset.

Why it matters

Billions of daily messages go through AI chatbots, many for ideation and writing, so if outputs from different AI systems increasingly converge, the diversity of ideas people are exposed to could shrink over time. This raises a real question about how much AI can be trusted as a creative partner going forward.

Figure A2: OLS Linearity, Normality, Independence, and Influential Observations
Figure A2: OLS Linearity, Normality, Independence, and Influential Observations

Terms in this paper

  • Alternate Uses Task (AUT) · A psychology test measuring divergent thinking by asking people (or models) to list unusual uses for common objects
  • Infinity-Chat100 · A dataset of 100 real-world open-ended questions people asked AI chatbots
  • sentence embedding · A way of turning text into a numerical vector that captures its meaning, so it can be compared mathematically
  • cosine distance · A measure of how different two vectors (here, two responses) are in meaning; lower means more similar
  • cross-family comparison · Comparing models from different companies rather than models made by the same company

Original abstract (English)

Many benchmarks track Large Language Model (LLM) performance on tasks with verifiable answers, but less is known about how LLM performance is evolving on open-ended tasks, where creativity, originality and diversity may matter as much as quality. As LLMs increasingly support human ideation and creative work, understanding trends in LLM performance on open-ended tasks is critical. This paper presents a preliminary analysis of LLM creative outputs spanning three years of model releases, examining model responses to Infinity-Chat100, a real-world collection of open-ended user queries, and the Alternate Uses Task, an established psychometric creativity assessment. Using sentence-embedding similarity, we examine trends in LLM responses to these prompts. Our findings show a statistically significant decrease in model output diversity over time, suggesting that LLM outputs may be converging in creative substance across models. If this trend persists, LLM-driven homogenization may progressively diminish human agency in human-AI co-creative work, demanding careful consideration of LLMs' role in the human creative process.

Authors · Nirav Patel, Josiah Crossman, Eva Aggarwal, Emily Wenger

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Nirav Patel et al., arXiv:2608.19437, CC BY 4.0