Chain-of-Experience for Continual LLM Improvement

arXiv:2608.180272026-08-17

LLMs can learn from their own mistakes mid-task, boosting accuracy while cutting API costs

This paper introduces Chain-of-Experience (CoE), a way for LLMs to keep reattempting a problem while accumulating feedback from previous tries, instead of answering once and stopping. Tested on 8 models including GPT-5, Gemini-2.5 Pro, and Claude-4.5 Sonnet across math, coding, and knowledge tasks, CoE consistently beat feedback-free baselines, delivering a 5.6% overall accuracy gain with 19% lower API cost. Even feedback generated by the model itself, without any ground-truth signal, produced large gains, and combining feedback types helped further.

METAL MEDIA explanatory visual

LLMs can learn from their own mistakes mid-task, boosting accuracy while cutting API costs

01Standard LLM evaluation treats every question as a one-shot event, ignoring whether models can actually get better by interacting with feedback over multiple tries -- this paper studies that ability directly
02CoE lets a model condition its next answer on the full history of prior attempts and the feedback each attempt received. Four feedback types were tested: none, execution feedback (code test results), model feedback (a judge model's critique), and correctness feedback (right/wrong signal)
03Experiments covered 8 state-of-the-art reasoning models (GPT-5, o3, Gemini-2.5 Pro, Claude-4.5 Sonnet, among others) on six benchmarks: AIME 2025, OmniMath (math), LiveCodeBench V6, LiveBench Code (coding), and GPQA Diamond (knowledge)
04Self feedback alone outperformed established test-time methods like few-shot in-context learning, Dynamic CheatSheet, and Agentic Context Engineering by 7-9%, while the strongest feedback setting pushed average accuracy to 79.3% versus roughly 62-67% for those baselines
05Models with stronger baseline (zero-shot) ability tended to improve more from feedback (average Pearson correlation of 0.5), and models stayed fairly robust even when fed deliberately wrong or misleading feedback

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Standard LLM evaluation treats every question as a one-shot event, ignoring whether models can actually get better by interacting with feedback over multiple tries -- this paper studies that ability directly
CoE lets a model condition its next answer on the full history of prior attempts and the feedback each attempt received. Four feedback types were tested: none, execution feedback (code test results), model feedback (a judge model's critique), and correctness feedback (right/wrong signal)
Experiments covered 8 state-of-the-art reasoning models (GPT-5, o3, Gemini-2.5 Pro, Claude-4.5 Sonnet, among others) on six benchmarks: AIME 2025, OmniMath (math), LiveCodeBench V6, LiveBench Code (coding), and GPQA Diamond (knowledge)
Self feedback alone outperformed established test-time methods like few-shot in-context learning, Dynamic CheatSheet, and Agentic Context Engineering by 7-9%, while the strongest feedback setting pushed average accuracy to 79.3% versus roughly 62-67% for those baselines
Models with stronger baseline (zero-shot) ability tended to improve more from feedback (average Pearson correlation of 0.5), and models stayed fairly robust even when fed deliberately wrong or misleading feedback

Figure 1: Summarized results on four benchmarks across math, code, and knowledge over four LLMs. Iterative problem solving under CoE provides three benefits: upper: by incorporating feedback in CoE, the average performance across four LLMs outperforms other test-time augmentations; lower left: models are capable of achieving the better performance with lower API cost with feedback; lower right: LLMs that perform better on the task display better improving capabilities through CoE with moderate to strong Pearson correlation. We present more explanations regarding baselines in Section 4.1.

Figure 2: An overview of our studies iterative improvement loop for LMs. The model ℳ learns by repeatedly interacting with the environment ℰ (e.g., model simulation or coding environment). In each cycle, ℳ generates actions conditioned on the input and accumulated experience, then receives feedback from ℰ to form a new experience. We investigate four specific feedback in this study: none, execution, model, and correctness feedback.

Table 1: Dual feedback and principled experience selection results using Claude 4.5 Sonnet. Dual feedback combines model feedback with correctness (math) or executor (code) signals. Memory-based methods (DC, SimpleMem) are applied within-task with no cross-task leakage. Acc: best accuracy (%) over 20 iterations; Best R: iteration achieving best performance.
	AIME 2025	LiveBench (Code)	OmniMath
Setting	Acc	Best R	Acc	Best R	Acc	Best R
Dual (Model + Corr/Exec)	76.7	R19	81.2	R15	73.5	R17
Correctness / Executor	70.0	R13	78.1	R15	74.5	R17
Binary Executor	–	–	71.9	R13	–	–
Model	60.0	R6	57.8	R17	50.5	R9
Model + DC	50.0	R8	51.6	R15	46.0	R10
Model + SimpleMem	56.7	R6	54.7	R17	49.5	R12

Figure 3: Results of five state-of-the-art LLMs on six benchmarks using different generation techniques. Models under CoE with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies (no feedback, DC, ICL, and the ones with different reasoning efforts). Results are averaged over 3 runs and we shade the standard deviation with a lighter color and plot bars showing model performance under different reasoning efforts. The full results are in Appendix H.

Figure 4: Total cost of each model over task completion vs. its best performance within 20 iterations. LLMs with feedback generally achieve higher scores with fewer costs (at the upper left), while iterative experience without feedback generally falls behind (at lower right). We provide full results over six benchmarks in Appendix H.

Table 2: The best performance over 20 iterations under constant correct” or incorrect” feedback (e.g., “the answer is correct”). Selective majority voting (SelMV) helps LLMs maintain performance. Results are averaged over 3 runs with best scores emphasized.
Feedback	AIME 2025	GPQA Diamond
	GPT-5 mini	o4 mini	GPT-5 mini	o4 mini
Self	93.3	91.1	79.9	78.8
SelMV Self	91.1	88.9	80.4	79.5
All Correct	90.0	73.3	79.3	75.8
SelMV Correct	93.3	73.3	79.3	76.3
Incorrect	91.7	83.3	79.3	72.7
SelMV Incorrect	89.7	86.7	82.8	77.8

Figure 5: Zero-shot performance of models (Base Capacity) and the learning gain show positive Pearson correlations (r), indicating that better-performing LLMs possess greater improving capability. Scores are averaged across 3 runs.

Figure 6: Percentages of different reasons for LLMs’ improvement patterns from 6,630 incorrect to correct response pairs. We employ GPT-5 to conduct this automatic analysis.

Table 3: Average performance comparison (%) across different LLMs on different datasets. For baselines, ICL, ACE, DC stands for few-shot in-context learning, agentic context engineering, and dynamic cheatsheet, respectively.
Method	AIME 2025	LiveCodeBench (V6)	LiveBench (Code)	OmniMath	GPQA Diamond	EvaLearn
ICL	71.83%	62.50%	65.46%	53.12%	78.45%	40.99%
ACE	71.98%	66.94%	69.38%	50.33%	76.58%	42.54%
DC	73.33%	63.59%	68.58%	48.64%	79.56%	42.68%
w/o Feedback	77.78%	72.57%	60.16%	65.17%	80.02%	44.91%
Reasoning-high	69.05%	70.63%	55.46%	61.81%	76.21%	39.58%
Reasoning-low	60.48%	61.03%	55.38%	50.60%	72.92%	29.34%
Binary-Executor	–	72.90%	71.65%	–	–	–
Self	82.22%	75.69%	69.94%	67.52%	81.03%	51.73%
Correctness/Executor	89.05%	74.50%	75.78%	79.61%	99.52%	57.05%

Figure 7: Model performance using constant “incorrect” and “correct” feedback. By leveraging the selective majority voting, LLMs show decent performance when facing spurious feedback on math tasks.

Figure 8: Performance of GPT-5 mini, o4-mini, and o3-mini with extended iterations of experience to 50 on math tasks. We mark performance points within the first 20 iterations of experience in blue, and those from iterations 20 to 50 in green.

Table 4: Cohen’s Kappa agreement between GPT-5 judge and human annotators across four improvement categories on 100 sampled trajectories. The overall κ of 0.768 indicates substantial agreement [landis1977measurement].
Category	Agreement (%)	Cohen’s κ
Feedback Fidelity	84.0	0.81
Self Reflection	72.0	0.71
Specification Recall	80.0	0.78
Random	68.0	0.63
Overall	76.0	0.768

Figure 9: Model performance of GPT-5 and GPT-5 mini with external model feedback on two math tasks.

Figure 10: On BrowseComp-Plus, self-feedback models fall behind as the task requires external search-based knowledge.

Table 5: Token complexity vs. accuracy across methods. CEF: Correctness/Executor Feedback; SF: Self Feedback; NF: No Feedback; DC: Dynamic CheatSheet. Token counts are aggregated across all iterations.
Dataset	Method	Tokens	Acc (%)
AIME 2025	CEF	108,734	84.6
SF	108,231	83.8
NF	106,825	74.1
DC	11,233	74.7
OmniMath	CEF	176,412	74.2
SF	175,806	72.1
NF	173,944	66.8
DC	16,904	63.9
LiveCodeBench (V6)	CEF	224,118	72.6
SF	223,441	71.2
NF	221,550	68.0
DC	20,771	66.4

Figure 11: Full results of eight state-of-the-art LLMs on six benchmarks incorporating different generation techniques. Model accuracies with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies.

Figure 12: Full results of total API cost (in dollar) vs. best model performance over 20 iterations. LLMs with detailed feedback (i.e., self feedback) achieves decent results with fewer costs, while CoE without feedback generally falls behind (at lower right).

Why it matters

It shows that instead of retraining a model, developers can get meaningful accuracy gains and lower inference costs simply by letting an LLM iterate on its own attempts with feedback -- relevant for anyone building multi-turn coding assistants, tutoring tools, or agents. It also gives practical guidance on which feedback type to invest in depending on task type and budget.

Terms in this paper

Chain-of-Experience (CoE) · A test-time method where a model repeatedly retries a task while accumulating its past attempts and the feedback each attempt received
self feedback · Feedback generated by a model judging its own (or another model's) answer, without access to the ground-truth answer
correctness feedback · A feedback signal that directly tells the model whether its answer was right or wrong
in-context learning (ICL) · A prompting method that shows the model a few solved examples before asking it to answer a new question
Dynamic CheatSheet / Agentic Context Engineering · Prior methods that build an external memory of reusable strategies distilled from previously solved problems

Original abstract (English)

Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inference-time interaction. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero-shot inference. We instantiate CoE with diverse feedback mechanisms, including model self-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet. Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines, achieving substantial gains with self feedback alone, alongside a 5.6% overall improvement and 19% lower API cost across tasks and models. We further show that combining complementary feedback channels (e.g., model and correctness signals) yields additional gains, and that CoE delivers higher accuracy per token than existing test-time strategies. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations.

Authors · Haoqin Tu

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

NCT 127 drop BLINGY, their first full album in two years

Figures: Haoqin Tu et al., arXiv:2608.18027, CC BY 4.0