Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
Demystifying Agent Skills: Why They Work-Until They Don't
arXiv:2608.140362026-08-13
AI agent 'skill' documents help not by teaching new facts but by stabilizing what the agent does step by step
This study dissects when, why, and where 'skills'-structured procedural documents given to LLM agents like SKILL.md-actually help, using 8,135 trial records and a hand-coded taxonomy from 240 sampled trajectories. The core finding is that skills mainly work as procedural anchors that stabilize repeated setup, tool-use, and verification steps, not by injecting missing knowledge. It also shows that as skill libraries grow, the precision of actually retrieving and using the right skill collapses sharply, yet downstream task success does not collapse in the same way.
METAL MEDIA explanatory visual
Skill vs Workflow Memory vs Raw: same experience, different representation
Evidence statusMeasured results reported
1. Collect raw trajectoriesSuccessful and failed executions for each task are gathered in a fixed Docker environment to form a balanced shared trace pool
2. Distill into three formsThe same pool is kept unprocessed (Raw), cleaned into procedural traces (Workflow Memory), or compressed into a standardized SKILL.md (Skill)
3. Re-run under matched protocolAll three conditions are evaluated on the same target tasks under identical execution rules
4. Contrastive taxonomy labelingPaired triples across the three arms are labeled by humans and an LLM judge into 12 fine-grained skill-use modes to reveal what changed
5. Retrieval-difficulty experimentsPool size and distractor type are varied across embedding retrieval, explicit selection, and real execution arms to separate retrieval quality from downstream success
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
The same underlying trajectory pool was distilled into three forms-unprocessed Raw, cleaned Workflow Memory, and a standardized SKILL.md-and evaluated on matched tasks to isolate the effect of representation alone
240 sampled trajectories were manually open-coded, yielding 238 valid unique labels consolidated into a taxonomy of 3 high-level categories and 12 fine-grained skill-use modes, validated against human judgment with 95.8% exact agreement (Cohen's kappa=0.952)
The same protocol was run across different agent-model pairings, including Codex + GPT-5.3-Codex and Gemini CLI + Gemini-3.1-Pro-Preview, to check whether findings held beyond a single model
Skill-retrieval quality was measured separately across three arms-embedding-based ranking, explicit agent selection without execution, and full real execution-while varying candidate pool size (5 to 100) and distractor type
Figure 1: Experimental pipelines. Top: skill versus procedural memory. We execute each task in a fixed Docker environment, collect successful and failed trajectories, and form a fixed-budget composition grid. The same trace pool is distilled either into Workflow Memory or into a reusable SKILL.md, which are then evaluated on matched tasks under the same protocol. Bottom: skill retrieval. Each task is paired with a candidate pool containing its ground-truth skill and k−1 real distractors (random, similar, or dissimilar). Matched pools are evaluated independently in three procedures: (A) embedding-based ranking without task execution, (B) explicit agent selection without Docker execution or verification, and (C) full-pool real execution with skill-use parsing after task verification. Outputs from (A) and (B) are not passed to (C).
Table 1: Task success rates for Workflow Memory and Skill injection across trajectory mixtures. Gray rows denote Raw baselines. Green, red, and unshaded cells indicate values above, below, and equal to the corresponding Raw baseline, respectively; bold marks the row-wise maximum across mixture settings. Mixture labels denote the numbers of successful (s) and failed (f) source trajectories. Terminal-Bench-Pro rates use 130 trials per condition, with infrastructure or verifier errors counted as failures.
Agent + Model
Trajectory Mix
Terminal-Bench-2
SkillsBench
Terminal-Bench-Pro
Workflow
Skill
Workflow
Skill
Workflow
Skill
Codex + GPT-5.3-Codex Raw
0.5935
0.5083
0.5394
Codex GPT-5.3-Codex
5s0f
0.4452
0.7548
0.5250
0.7250
0.7333
0.7455
4s1f
0.4000
0.7290
0.5667
0.6167
0.7333
0.7939
3s2f
0.4194
0.7806
0.6417
0.6250
0.6970
0.7333
2s3f
0.3677
0.6839
0.6083
0.7083
0.6667
0.6667
1s4f
0.2710
0.7097
0.5167
0.6167
0.6121
0.5818
0s5f
0.2839
0.5161
0.5833
0.4500
0.4788
0.4303
Gemini CLI + Gemini-3.1-Pro-Preview Raw
0.5000
0.4762
0.5615
Gemini CLI Gemini-3.1-Pro-Preview
5s0f
0.6231
0.7923
0.5524
0.7429
0.5308
0.6692
4s1f
0.5308
0.7615
0.5238
0.6190
0.6923
0.6308
3s2f
0.6462
0.7462
0.5619
0.6667
0.6462
0.5462
2s3f
0.6000
0.7000
0.5524
0.6762
0.6846
0.5077
1s4f
0.5846
0.6923
0.4857
0.6000
0.5923
0.5692
0s5f
0.5231
0.4769
0.4286
0.4095
0.4769
0.4615
(b) Three-experiment skill-retrieval evaluation.
Table 2: Mechanism labels used to characterize how injected prior experience affects execution.
Mechanism
Meaning
procedural_anchor
The artifact gives a usable procedure, ordering, checklist, tool sequence, or verification plan.
knowledge_ injection
The artifact supplies concrete domain knowledge that the agent otherwise lacked.
failure_warning
The artifact warns about a pitfall that the agent avoids.
none
The artifact is not used in a meaningful way.
counterproductive
The artifact misleads the agent or makes the run worse.
Figure 2: Taxonomy label distribution across trajectory mixtures and experimental arms. Stacked bars show trajectory-level labels for Raw, Workflow Memory, and Skill across the six source-trajectory mixtures from 5s0f to 0s5f. Labels are grouped into three high-level categories; per-mode percentages are reported in Appendix Table 11.
Table 3: Human validation of the taxonomy construction pipeline. The first stage checks whether raw labels are supported by their source trajectories; the second independently maps those labels to the 12 canonical modes.
Validation stage
Evaluation units
Result
Trajectory grounding
714 checks (238 labels × 3 trajectories)
All labels confirmed
Taxonomy aggregation
238 valid unique labels
95.8% exact; Cohen’s κ=0.952
Figure 3: Skill retrieval and execution-time skill use on SkillsBench. Left: precision for the two offline diagnostics. Right: parsed actual-use precision (solid lines) and downstream success (dashed lines) for Arm 3. Curves average over random, similar, and dissimilar pool regimes; outputs are not passed between experiments.
Table 4: Effect of pool composition and size on SkillsBench. Entries report percentages for the indicated retrieval arm, pool composition, and pool size. Arms 2 and 3 are arithmetic means over the reported agent–model pairings. Full recall and F1 values are given in Appendix Table 15.
Pool / metric
Skill-pool size k
5
10
20
50
100
Random
Arm 1 P
97.7
95.5
95.5
92.0
84.1
Arm 2 P
78.1
77.9
82.1
76.5
69.8
Arm 3 P
25.9
23.2
19.5
8.6
4.4
Arm 3 Succ.
31.8
36.8
40.1
36.3
41.9
Similar
Arm 1 P
70.5
63.6
60.2
56.8
53.4
Arm 2 P
53.1
52.9
47.1
48.6
43.7
Arm 3 P
34.5
22.3
15.7
7.3
3.7
Arm 3 Succ.
41.7
39.6
39.2
39.5
39.6
Dissimilar
Arm 1 P
96.6
96.6
96.6
94.3
93.2
Arm 2 P
78.9
81.6
82.6
78.4
77.8
Arm 3 P
28.6
19.2
9.0
4.4
1.7
Arm 3 Succ.
35.7
36.9
33.7
38.8
36.4
(b) Arm 3 skill-use precision and downstream success.
Table 5: Experimental arms in the contrastive trajectory analysis.
Arm
Injected prior experience
Purpose
Raw
No injected prior trajectory or skill
Baseline behavior of the agent on the task.
Workflow memory
Cleaned prior workflows are appended as procedural memory
Tests whether direct trajectory-like procedural memory improves execution.
Skill
The same prior workflows are distilled into a standardized reusable skill
Tests whether compact skill representation improves over direct workflow memory.
Figure 4: Cross-framework transfer of procedural experience. Prior-experience artifacts constructed in one agent framework are evaluated in another. Dashed lines indicate the target framework’s Raw baseline.
Table 6: Input artifacts used by the taxonomy pipeline.
Stores the terminal/tool-use trajectory and the agent’s reasoning-visible interaction record.
Task instruction
Defines the task objective and, for workflow-memory arms, may include injected workflow content.
Skill artifact
Stores the injected skill used in the skill arm.
Task-side files
Used only as contextual artifacts when present; the main taxonomy labels are based on execution trajectories and verifier outcomes.
Figure 5: Effect of outcome labels during skill creation. Panels compare skills created with outcome labels visible (normal) or withheld (no-hint) across trajectory mixtures and benchmarks. Dashed lines indicate the corresponding Raw baselines. Terminal-Bench-Pro entries use 130 trials per condition; missing or infrastructure-error trials count as failures.
Table 7: Coverage statistics after manifest construction and artifact linking.
Dimension
Count
Terminal-Bench 2.0 trials
3,254
Terminal-Bench-Pro trials
2,993
SkillsBench trials
1,888
Raw-arm trials
1,883
Workflow-memory trials
2,658
Skill-arm trials
3,594
Successful trials
4,541
Failed trials
3,594
Records with available agent transcript
7,837
Records with available task instruction
6,210
Skill-arm records with linked skill artifact
3,570
Table 8: Paired triple sample used for contrastive taxonomy labeling.
Split
Count
SkillsBench triples
144
Terminal-Bench 2.0 triples
186
Terminal-Bench-Pro triples
198
Triples per mixture setting
88
Total triples
528
Table 9: Oracle-status success rates across the three execution arms.
Arm
Success / total
Success rate
Raw
312 / 528
59.1%
Workflow memory
295 / 528
55.9%
Skill
327 / 528
61.9%
Table 10: Paired success-rate deltas between execution arms.
Comparison
Mean paired delta
95% bootstrap CI
WM vs Raw
−0.0322
[−0.0814,+0.0208]
Skill vs Raw
+0.0284
[−0.0227,+0.0795]
Skill vs WM
+0.0606
[+0.0076,+0.1136]
Table 11: Contrastive skill-use taxonomy over 528 paired triples. SC abbreviates Skill-use Category, the top-level taxonomy label assigned to a trajectory; each SC groups the fine-grained modes listed in the table. Percentages are computed within each arm over the same paired-triple sample. SC1 denotes successful procedural anchoring, SC2 execution-layer and verification failures, and SC3 invocation, applicability, and boundary failures.
SC
Mode
Raw
WF
Skill
SC1
skill_guided_success
10.4%
0.4%
61.6%
SC1
workflow_guided_success
0.0%
54.5%
0.0%
SC1
autonomous_clean_success
48.7%
0.8%
0.2%
SC2
environment_infrastructure_failure
5.3%
1.7%
0.2%
SC2
output_format_schema_mismatch
7.4%
3.8%
3.2%
SC2
background_service_lifecycle_failure
2.7%
2.5%
0.8%
SC2
shell_code_corruption
1.1%
1.9%
0.2%
SC2
algorithmic_logic_error
8.3%
11.0%
7.4%
SC2
static_verification_without_runtime
12.5%
12.5%
11.7%
SC3
timeout_budget_exhaustion
1.7%
10.6%
4.4%
SC3
skill_guidance_misapplied_or_ignored
0.8%
0.4%
10.0%
SC3
capability_or_safety_limit
1.1%
0.0%
0.4%
Table 12: Lightweight compact procedural baselines on selected Terminal-Bench-2 tasks. Entries report downstream success for Raw, short-plan, test-first, Workflow Memory, and Skill conditions over 26 tasks with five trials per task.
Condition
Source
Success / total
Success rate
Raw
None
65 / 130
50.0%
Short plan
Task instruction
62 / 130
47.7%
Test-first template
Workflow
77 / 130
59.2%
Workflow Memory
Workflow
81 / 130
62.3%
Skill
Workflow
103 / 130
79.2%
Table 13: Matched success and token-cost comparison. Entries are computed on the 83-task intersection with equal task weighting. Token counts are per-task averages reported in thousands (K); “pp” denotes percentage points.
Absolute metrics on the matched 83-task intersection
Representation
Success
Input
Output
Total
Δ succ. vs Raw
Cost profile
Raw trajectories
64.1%
541.5K
14.2K
555.7K
–
Full prior traces provide broad evidence but carry the largest context load.
Workflow Memory
64.8%
417.9K
8.3K
426.2K
+0.7 pp
Most token-efficient representation after cleaning trajectory noise.
Skill
69.6%
511.7K
9.8K
521.5K
+5.5 pp
Highest success rate, with lower token use than Raw but higher token use than Workflow Memory.
Pairwise trade-offs
Comparison
Δ success
Δ input
Δ output
Δ total
Direction
Interpretation
Workflow Memory vs Raw
+0.7 pp
-123.6K
-5.9K
-129.5K
cheaper
Workflow Memory substantially reduces token cost with nearly unchanged success.
Skill vs Raw
+5.5 pp
-29.8K
-4.4K
-34.2K
better and cheaper
Skill improves success while still reducing token use relative to Raw trajectories.
Skill vs Workflow Memory
+4.8 pp
+93.8K
+1.5K
+95.3K
better but costlier
Skill trades additional context for stronger execution performance.
Table 14: Complete Arm 1 embedding-retrieval results on SkillsBench. Entries report ranking metrics from Qwen3-Embedding-0.6B using task–skill-description similarity. Top-5 is omitted for k=5 because it covers the full candidate pool.
Pool
k
Top-1
Top-3
Top-5
P
P
R
F1
P
R
F1
Random
5
97.7
33.0
98.9
49.4
–
–
–
10
95.5
32.6
97.7
48.9
19.5
97.7
32.6
20
95.5
32.2
96.6
48.3
19.5
97.7
32.6
50
92.0
32.2
96.6
48.3
19.5
97.7
32.6
100
84.1
30.7
92.0
46.0
19.1
95.5
31.8
Similar
5
70.5
31.8
95.5
47.7
–
–
–
10
63.6
29.2
87.5
43.8
19.3
96.6
32.2
20
60.2
26.9
80.7
40.3
17.5
87.5
29.2
50
56.8
24.2
72.7
36.4
16.4
81.8
27.3
100
53.4
22.7
68.2
34.1
15.7
78.4
26.1
Dissimilar
5
96.6
33.0
98.9
49.4
–
–
–
10
96.6
32.2
96.6
48.3
19.8
98.9
33.0
20
96.6
32.2
96.6
48.3
19.5
97.7
32.6
50
94.3
32.2
96.6
48.3
19.5
97.7
32.6
100
93.2
32.2
96.6
48.3
19.3
96.6
32.2
Table 15: Complete Arm 2 and Arm 3 retrieval results on SkillsBench. Rows correspond to agent–model, distractor regime, and pool size. Arm 2 reports explicit-selection precision, recall, and F1; Arm 3 reports parsed actual-use precision, recall, F1, and downstream success. Dashes indicate excluded entries.
Agent / Model
Pool
k
Arm 2: Agent Selection
Arm 3: Real Execution
P
R
F1
P
R
F1
Succ.
Gemini CLI Gemini-3.1-Pro-Preview
Random
5
74.4
75.0
74.7
18.6
69.4
29.3
39.2
Random
10
72.7
72.7
72.7
7.1
66.4
12.9
38.1
Random
20
80.1
81.8
81.0
3.6
66.0
6.8
39.2
Random
50
81.2
84.1
82.6
1.4
65.8
2.8
37.6
Random
100
77.4
83.0
80.1
0.7
66.0
1.4
39.0
Similar
5
54.3
69.3
60.9
17.6
70.1
28.1
38.8
Similar
10
61.3
79.5
69.2
7.0
67.3
12.6
38.5
Similar
20
58.4
77.3
66.6
3.7
66.2
6.9
34.0
Similar
50
59.3
76.1
66.7
1.4
66.2
2.7
36.7
Similar
100
55.4
73.9
63.3
0.7
66.7
1.4
36.1
Dissimilar
5
74.4
76.1
75.3
14.6
63.2
23.7
34.0
Dissimilar
10
80.1
81.8
81.0
8.7
66.0
15.3
38.8
Dissimilar
20
82.4
83.0
82.7
5.1
65.5
9.5
35.6
Dissimilar
50
79.3
80.7
80.0
1.6
64.4
3.1
38.1
Dissimilar
100
83.5
84.1
83.8
0.7
61.6
1.5
34.5
Codex GPT-5.4
Random
5
81.8
84.1
82.9
33.1
41.6
36.8
24.3
Random
10
83.0
86.4
84.6
39.2
59.7
47.3
35.5
Random
20
84.1
87.5
85.8
35.3
69.3
46.8
40.9
Random
50
71.7
87.5
78.8
15.8
63.5
25.4
35.0
Random
100
62.2
85.2
71.9
8.1
73.6
14.6
44.8
Similar
5
51.9
81.8
63.5
51.3
72.4
60.0
44.5
Similar
10
44.4
83.0
57.8
37.5
66.0
47.9
40.7
Similar
20
35.8
77.3
48.9
27.7
66.1
39.1
44.3
Similar
50
37.8
76.1
50.5
13.1
61.8
21.6
42.3
Similar
100
31.9
70.5
43.9
6.7
54.3
11.9
43.0
Dissimilar
5
83.3
85.2
84.3
42.5
63.0
50.7
37.3
Dissimilar
10
83.0
85.2
84.1
29.7
61.3
40.0
35.0
Dissimilar
20
82.8
85.2
84.0
12.7
60.5
21.0
31.8
Dissimilar
50
77.5
85.2
81.2
7.2
65.1
12.9
39.5
Table 16: Complete numerical results for the outcome-annotation ablation. Entries report downstream success for skills constructed under the indicated trajectory mixture. normal exposes source-trajectory outcomes during construction; no-hint withholds them. Terminal-Bench-Pro entries use 130 trials per condition, with missing or infrastructure-error trials counted as failures.
Agent + Model
Benchmark
Creator
5s0f
4s1f
3s2f
2s3f
1s4f
0s5f
Codex GPT-5.3-Codex
TB2
normal
0.7548
0.7290
0.7806
0.6839
0.7097
0.5161
no-hint
0.7677
0.7355
0.5871
0.4968
0.5548
0.3871
SB
normal
0.7250
0.6167
0.6250
0.7083
0.6167
0.4500
no-hint
0.6667
0.6417
0.5583
0.5000
0.5083
0.3500
TB-Pro
normal
0.7455
0.7939
0.7333
0.6667
0.5818
0.4303
no-hint
0.8364
0.6606
0.5758
0.5152
0.4848
0.3758
Gemini CLI Gemini-3.1-Pro-Preview
TB2
normal
0.7923
0.7615
0.7462
0.7000
0.6923
0.4769
no-hint
0.4231
0.4923
0.4000
0.3692
0.5231
0.4308
SB
normal
0.7429
0.6190
0.6667
0.6762
0.6000
0.4095
no-hint
0.6190
0.5143
0.4095
0.4190
0.4095
0.4000
TB-Pro
normal
0.6692
0.6308
0.5462
0.5077
0.5692
0.4615
no-hint
0.6154
0.5769
0.6385
0.5923
0.5692
0.5154
Findings
In matched comparisons, Skill outperformed Workflow Memory by 6.06 percentage points (95% bootstrap CI [+0.76, +11.36])
Among skill mechanisms, procedural_anchor accounted for 65.7% of cases versus only 4.5% for explicit knowledge_injection
As candidate pool size grew from 5 to 100, actual-use precision during real execution fell from 29.6% to 3.3%
Confusable distractor skills degraded offline identification accuracy, but downstream task success remained largely stable
Oracle-status success rates were 61.9% for Skill, 59.1% for Raw, and 55.9% for Workflow Memory; SC2 execution-layer/verification failures dropped from 37.3% (Raw) and 33.3% (Workflow) to 23.5% (Skill), while SC3 invocation/boundary failures rose sharply relative to Raw (78/528 vs 19/528)
Where it can be used
Designing reusable SKILL.md procedural guides for coding or terminal-operating LLM agents, informed by what actually made skills effective in these tests
Considering distillation into standardized procedural artifacts, rather than raw trace accumulation, when building self-evolving agents that learn from experience
Separating retrieval-quality evaluation from execution-quality evaluation when maintaining a growing skill library, given the observed retrieval-precision drop-off
Limits and open work
Experiments were run on terminal- and software-engineering-oriented benchmarks (Terminal-Bench, Terminal-Bench-Pro, SkillsBench) with specific agent-model pairings (Codex + GPT-5.3-Codex, Gemini CLI + Gemini-3.1-Pro-Preview), so generalization to other domains is not established
RQ4 used GPT-5.4 instead of GPT-5.3-Codex for availability reasons, so its results should be read only as within-pairing comparisons, not directly compared in absolute terms with RQ1-RQ3
Failure modes like algorithmic_logic_error and static_verification_without_runtime remained substantial across Skill, Workflow Memory, and Raw arms, meaning skills do not reliably fix wrong logic or replace stronger runtime verification
Skills introduced a new failure surface (SC3: invocation, applicability, and boundary failures) where guidance was misapplied, over-applied, or ignored, requiring separate scrutiny when adopting skills
Why it matters
Skill evaluations have mostly relied on aggregate task success, which hides why a skill helped or failed; this work breaks that black box into concrete behavioral mechanisms. For anyone building self-improving agents or maintaining skill libraries, it offers grounded evidence on when skill guidance actually pays off and where it introduces new failure modes.
Terms in this paper
Skill · A compact, standardized procedural document (SKILL.md) describing what to do, what to check, and what pitfalls to avoid, distilled from prior execution traces
Workflow Memory · A cleaned but trace-level record of past execution that preserves procedural flow more directly than a distilled skill
Procedural anchoring · The mechanism by which a skill stabilizes concrete actions-setup steps, tool sequences, checks-rather than supplying new factual knowledge
Open coding · A qualitative method where human annotators read trajectories and freely describe observed success/failure patterns before categorization
Oracle-status success rate · Task success rate as determined by the benchmark's verifier, used as the primary aggregate outcome metric
Original abstract (English)
Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: \textbf{When do skills help, why do they work, and where do they fail?} Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7\% of skill cases, versus 4.5\% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts. Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6\% to 3.3\%. Confusable distractors impair offline identification, yet downstream success remains stable; exact ground-truth invocation is neither sufficient nor necessary. Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation. These findings move evaluation beyond aggregate success rates and guide reliable self-evolving agents.
