Query Timing Produces Opposite Positional Biases Between LLMs and Humans
AI judges show the opposite bias from humans depending on when you ask them to decide
Earlier research found that humans show a recency bias -- being swayed by the last evidence they hear -- only when asked for their opinion after every piece of evidence, not when asked just once at the end. This paper runs the same kind of experiment on LLMs like GPT and Claude and finds the reverse pattern. LLMs show little bias when questioned step by step, but a strong recency bias when questioned only at the end, and this bias gets worse in newer model versions.
METAL MEDIA explanatory visual
AI judges show the opposite bias from humans depending on when you ask them to decide
- 01The researchers built three mock accusation scenarios -- a criminal case, academic misconduct, and workplace misconduct -- each with four pieces of prosecution evidence and four pieces of defense evidence, and tested GPT, Claude, and open-source models by varying the order in which evidence was shown
- 02They compared two response modes: Step-by-Step (SbS), where the model answers questions after each piece of evidence, and End-of-Sequence (EoS), where it only answers after seeing all evidence from one side; each condition was run 30 times to measure the proportion of guilty verdicts
- 03Statistical tests showed that for most models, evidence order made no significant difference under SbS, but under EoS there was a clear recency bias -- models favored whichever side's evidence came last
- 04Older models like GPT 3.5 Turbo and Claude 3 Haiku showed no significant bias in either mode, while newer models like GPT 4o, Claude 3.7 Sonnet, and Claude 4 Sonnet showed a stronger recency bias specifically under the EoS mode
- 05Gemini 2.5 Flash was an exception with no significant bias in either mode, and open-source models showed less consistent patterns than GPT and Claude
What they did
- The researchers built three mock accusation scenarios -- a criminal case, academic misconduct, and workplace misconduct -- each with four pieces of prosecution evidence and four pieces of defense evidence, and tested GPT, Claude, and open-source models by varying the order in which evidence was shown
- They compared two response modes: Step-by-Step (SbS), where the model answers questions after each piece of evidence, and End-of-Sequence (EoS), where it only answers after seeing all evidence from one side; each condition was run 30 times to measure the proportion of guilty verdicts
- Statistical tests showed that for most models, evidence order made no significant difference under SbS, but under EoS there was a clear recency bias -- models favored whichever side's evidence came last
- Older models like GPT 3.5 Turbo and Claude 3 Haiku showed no significant bias in either mode, while newer models like GPT 4o, Claude 3.7 Sonnet, and Claude 4 Sonnet showed a stronger recency bias specifically under the EoS mode
- Gemini 2.5 Flash was an exception with no significant bias in either mode, and open-source models showed less consistent patterns than GPT and Claude
| EoS | SbS | |||
|---|---|---|---|---|
| Model | DP | PD | DP | PD |
| GPT 4o * | 1.00 | 0.033 | 0.30 | 0.033 |
| Claude 3.7 Sonnet * | 1.00 | 0.433 | 0.967 | 0.833 |
| Claude 4 Sonnet * | 1.00 | 0.20 | 1.00 | 0.967 |
| Gemini 2.5 Flash | 1.00 | 1.00 | 1.00 | 1.00 |
| Llama-4-Maverick | 1.00 | 0.8 | 0.6 | 0.033 |
| Qwen2.5-72b-Turbo * | 0.567 | 0.133 | 1.00 | 1.00 |
| EoS | SbS | |||
|---|---|---|---|---|
| Model | DP | PD | DP | PD |
| GPT 4o * | 0.2667 | 0.00 | 0.00 | 0.00 |
| Claude 3.7 Sonnet * | 0.967 | 0.00 | 0.1 | 0.00 |
| Claude 4 Sonnet * | 0.967 | 0.00 | 0.367 | 0.067 |
| Gemini 2.5 Flash | 0.567 | 0.467 | 0.30 | 0.33 |
| Llama-4-Maverick * | 0.967 | 0.00 | 0.167 | 0.00 |
| Qwen2.5-72b-Turbo * | 0.9 | 0.033 | 0.367 | 0.167 |
| EoS | SbS | |||
|---|---|---|---|---|
| Model | DP | PD | DP | PD |
| GPT 4o * | 1.00 | 0.20 | 1.00 | 0.867 |
| Claude 3.7 Sonnet * | 0.967 | 0.00 | 0.833 | 1.00 |
| Claude 4 Sonnet * | 0.967 | 0.00 | 0.933 | 0.867 |
| Gemini 2.5 Flash | 0.9 | 0.7 | 0.8 | 0.833 |
| Llama-4-Maverick * | 0.867 | 0.033 | 0.8 | 0.933 |
| Qwen2.5-72b-Turbo | 0.833 | 0.7 | 1.00 | 1.00 |
| Model | DP | PD |
|---|---|---|
| GPT 4o | 0.900 | 0.900 |
| Gemini 2.5 Flash | 0.667 | 0.600 |
| Claude 4 Sonnet (4-6) | 0.033 | 0.233 |
| Claude 4 Sonnet (4-20250514) | 1.000 | 0.400 |
| Meta Llama | 0.133 | 0.100 |
| Qwen | 1.000 | 1.000 |
Why it matters
As LLMs are increasingly used to make real decisions -- like judging cases, evaluations, or disputes -- the fact that simply changing when and how evidence is presented can flip the outcome is a serious reliability concern. It's also notable that this bias gets worse rather than better in newer, more capable models, meaning it won't just fix itself with model upgrades.
Terms in this paper
- positional bias · a tendency for judgments to shift depending on the order information is presented
- recency bias · being more influenced by the most recently presented information
- primacy bias · being more influenced by the first information presented
- SbS / EoS (Step-by-Step / End-of-Sequence) · answering after each piece of evidence versus answering only once at the very end
- LLM-as-a-judge · using a large language model to act as an evaluator or decision-maker in place of a human
Figures we cannot republish
- Figure 1: Changes in proportion of ”guilty” verdicts across GPT model family for criminal misconduct setting
- Figure 2: Changes in proportion of ”guilty” verdicts across Claude models for criminal misconduct setting
Original abstract (English)
Positional biases such as recency and primacy effects have been documented in large language models (LLMs), yet the underlying mechanism by which these models make their evaluations remains poorly understood. Both primacy and recency biases have been observed in human judgments in response to evidence, but recent work suggest that \emph{when} the listener updates their beliefs -- during the presentation of evidence or only at the end -- influences the presence of such effects. We investigate whether a similar phenomenon holds for LLMs, finding divergence from human behavior. These biases are more exacerbated in newer models compared to their predecessors.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctly
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears