Figure 1: Experimental pipelines. Top: skill versus procedural memory. We execute each task in a fixed Docker environment, collect successful and failed trajectories, and form a fixed-budget composition grid. The same trace pool is distilled either into Workflow Memory or into a reusable SKILL.md, which are then evaluated on matched tasks under the same protocol. Bottom: skill retrieval. Each task is paired with a candidate pool containing its ground-truth skill and k−1 real distractors (random, similar, or dissimilar). Matched pools are evaluated independently in three procedures: (A) embedding-based ranking without task execution, (B) explicit agent selection without Docker execution or verification, and (C) full-pool real execution with skill-use parsing after task verification. Outputs from (A) and (B) are not passed to (C).
Table 1: Task success rates for Workflow Memory and Skill injection across trajectory mixtures. Gray rows denote Raw baselines. Green, red, and unshaded cells indicate values above, below, and equal to the corresponding Raw baseline, respectively; bold marks the row-wise maximum across mixture settings. Mixture labels denote the numbers of successful (s) and failed (f) source trajectories. Terminal-Bench-Pro rates use 130 trials per condition, with infrastructure or verifier errors counted as failures.
Agent + Model
Trajectory Mix
Terminal-Bench-2
SkillsBench
Terminal-Bench-Pro
Workflow
Skill
Workflow
Skill
Workflow
Skill
Codex + GPT-5.3-Codex Raw
0.5935
0.5083
0.5394
Codex GPT-5.3-Codex
5s0f
0.4452
0.7548
0.5250
0.7250
0.7333
0.7455
4s1f
0.4000
0.7290
0.5667
0.6167
0.7333
0.7939
3s2f
0.4194
0.7806
0.6417
0.6250
0.6970
0.7333
2s3f
0.3677
0.6839
0.6083
0.7083
0.6667
0.6667
1s4f
0.2710
0.7097
0.5167
0.6167
0.6121
0.5818
0s5f
0.2839
0.5161
0.5833
0.4500
0.4788
0.4303
Gemini CLI + Gemini-3.1-Pro-Preview Raw
0.5000
0.4762
0.5615
Gemini CLI Gemini-3.1-Pro-Preview
5s0f
0.6231
0.7923
0.5524
0.7429
0.5308
0.6692
4s1f
0.5308
0.7615
0.5238
0.6190
0.6923
0.6308
3s2f
0.6462
0.7462
0.5619
0.6667
0.6462
0.5462
2s3f
0.6000
0.7000
0.5524
0.6762
0.6846
0.5077
1s4f
0.5846
0.6923
0.4857
0.6000
0.5923
0.5692
0s5f
0.5231
0.4769
0.4286
0.4095
0.4769
0.4615
(b) Three-experiment skill-retrieval evaluation.
Table 2: Mechanism labels used to characterize how injected prior experience affects execution.
Mechanism
Meaning
procedural_anchor
The artifact gives a usable procedure, ordering, checklist, tool sequence, or verification plan.
knowledge_ injection
The artifact supplies concrete domain knowledge that the agent otherwise lacked.
failure_warning
The artifact warns about a pitfall that the agent avoids.
none
The artifact is not used in a meaningful way.
counterproductive
The artifact misleads the agent or makes the run worse.
Figure 2: Taxonomy label distribution across trajectory mixtures and experimental arms. Stacked bars show trajectory-level labels for Raw, Workflow Memory, and Skill across the six source-trajectory mixtures from 5s0f to 0s5f. Labels are grouped into three high-level categories; per-mode percentages are reported in Appendix Table 11.
Table 3: Human validation of the taxonomy construction pipeline. The first stage checks whether raw labels are supported by their source trajectories; the second independently maps those labels to the 12 canonical modes.
Validation stage
Evaluation units
Result
Trajectory grounding
714 checks (238 labels × 3 trajectories)
All labels confirmed
Taxonomy aggregation
238 valid unique labels
95.8% exact; Cohen’s κ=0.952
Figure 3: Skill retrieval and execution-time skill use on SkillsBench. Left: precision for the two offline diagnostics. Right: parsed actual-use precision (solid lines) and downstream success (dashed lines) for Arm 3. Curves average over random, similar, and dissimilar pool regimes; outputs are not passed between experiments.
Table 4: Effect of pool composition and size on SkillsBench. Entries report percentages for the indicated retrieval arm, pool composition, and pool size. Arms 2 and 3 are arithmetic means over the reported agent–model pairings. Full recall and F1 values are given in Appendix Table 15.
Pool / metric
Skill-pool size k
5
10
20
50
100
Random
Arm 1 P
97.7
95.5
95.5
92.0
84.1
Arm 2 P
78.1
77.9
82.1
76.5
69.8
Arm 3 P
25.9
23.2
19.5
8.6
4.4
Arm 3 Succ.
31.8
36.8
40.1
36.3
41.9
Similar
Arm 1 P
70.5
63.6
60.2
56.8
53.4
Arm 2 P
53.1
52.9
47.1
48.6
43.7
Arm 3 P
34.5
22.3
15.7
7.3
3.7
Arm 3 Succ.
41.7
39.6
39.2
39.5
39.6
Dissimilar
Arm 1 P
96.6
96.6
96.6
94.3
93.2
Arm 2 P
78.9
81.6
82.6
78.4
77.8
Arm 3 P
28.6
19.2
9.0
4.4
1.7
Arm 3 Succ.
35.7
36.9
33.7
38.8
36.4
(b) Arm 3 skill-use precision and downstream success.
Table 5: Experimental arms in the contrastive trajectory analysis.
Arm
Injected prior experience
Purpose
Raw
No injected prior trajectory or skill
Baseline behavior of the agent on the task.
Workflow memory
Cleaned prior workflows are appended as procedural memory
Tests whether direct trajectory-like procedural memory improves execution.
Skill
The same prior workflows are distilled into a standardized reusable skill
Tests whether compact skill representation improves over direct workflow memory.
Figure 4: Cross-framework transfer of procedural experience. Prior-experience artifacts constructed in one agent framework are evaluated in another. Dashed lines indicate the target framework’s Raw baseline.
Table 6: Input artifacts used by the taxonomy pipeline.
Stores the terminal/tool-use trajectory and the agent’s reasoning-visible interaction record.
Task instruction
Defines the task objective and, for workflow-memory arms, may include injected workflow content.
Skill artifact
Stores the injected skill used in the skill arm.
Task-side files
Used only as contextual artifacts when present; the main taxonomy labels are based on execution trajectories and verifier outcomes.
Figure 5: Effect of outcome labels during skill creation. Panels compare skills created with outcome labels visible (normal) or withheld (no-hint) across trajectory mixtures and benchmarks. Dashed lines indicate the corresponding Raw baselines. Terminal-Bench-Pro entries use 130 trials per condition; missing or infrastructure-error trials count as failures.
Table 7: Coverage statistics after manifest construction and artifact linking.
Dimension
Count
Terminal-Bench 2.0 trials
3,254
Terminal-Bench-Pro trials
2,993
SkillsBench trials
1,888
Raw-arm trials
1,883
Workflow-memory trials
2,658
Skill-arm trials
3,594
Successful trials
4,541
Failed trials
3,594
Records with available agent transcript
7,837
Records with available task instruction
6,210
Skill-arm records with linked skill artifact
3,570
Table 8: Paired triple sample used for contrastive taxonomy labeling.
Split
Count
SkillsBench triples
144
Terminal-Bench 2.0 triples
186
Terminal-Bench-Pro triples
198
Triples per mixture setting
88
Total triples
528
Table 9: Oracle-status success rates across the three execution arms.
Arm
Success / total
Success rate
Raw
312 / 528
59.1%
Workflow memory
295 / 528
55.9%
Skill
327 / 528
61.9%
Table 10: Paired success-rate deltas between execution arms.
Comparison
Mean paired delta
95% bootstrap CI
WM vs Raw
−0.0322
[−0.0814,+0.0208]
Skill vs Raw
+0.0284
[−0.0227,+0.0795]
Skill vs WM
+0.0606
[+0.0076,+0.1136]
Table 11: Contrastive skill-use taxonomy over 528 paired triples. SC abbreviates Skill-use Category, the top-level taxonomy label assigned to a trajectory; each SC groups the fine-grained modes listed in the table. Percentages are computed within each arm over the same paired-triple sample. SC1 denotes successful procedural anchoring, SC2 execution-layer and verification failures, and SC3 invocation, applicability, and boundary failures.
SC
Mode
Raw
WF
Skill
SC1
skill_guided_success
10.4%
0.4%
61.6%
SC1
workflow_guided_success
0.0%
54.5%
0.0%
SC1
autonomous_clean_success
48.7%
0.8%
0.2%
SC2
environment_infrastructure_failure
5.3%
1.7%
0.2%
SC2
output_format_schema_mismatch
7.4%
3.8%
3.2%
SC2
background_service_lifecycle_failure
2.7%
2.5%
0.8%
SC2
shell_code_corruption
1.1%
1.9%
0.2%
SC2
algorithmic_logic_error
8.3%
11.0%
7.4%
SC2
static_verification_without_runtime
12.5%
12.5%
11.7%
SC3
timeout_budget_exhaustion
1.7%
10.6%
4.4%
SC3
skill_guidance_misapplied_or_ignored
0.8%
0.4%
10.0%
SC3
capability_or_safety_limit
1.1%
0.0%
0.4%
Table 12: Lightweight compact procedural baselines on selected Terminal-Bench-2 tasks. Entries report downstream success for Raw, short-plan, test-first, Workflow Memory, and Skill conditions over 26 tasks with five trials per task.
Condition
Source
Success / total
Success rate
Raw
None
65 / 130
50.0%
Short plan
Task instruction
62 / 130
47.7%
Test-first template
Workflow
77 / 130
59.2%
Workflow Memory
Workflow
81 / 130
62.3%
Skill
Workflow
103 / 130
79.2%
Table 13: Matched success and token-cost comparison. Entries are computed on the 83-task intersection with equal task weighting. Token counts are per-task averages reported in thousands (K); “pp” denotes percentage points.
Absolute metrics on the matched 83-task intersection
Representation
Success
Input
Output
Total
Δ succ. vs Raw
Cost profile
Raw trajectories
64.1%
541.5K
14.2K
555.7K
–
Full prior traces provide broad evidence but carry the largest context load.
Workflow Memory
64.8%
417.9K
8.3K
426.2K
+0.7 pp
Most token-efficient representation after cleaning trajectory noise.
Skill
69.6%
511.7K
9.8K
521.5K
+5.5 pp
Highest success rate, with lower token use than Raw but higher token use than Workflow Memory.
Pairwise trade-offs
Comparison
Δ success
Δ input
Δ output
Δ total
Direction
Interpretation
Workflow Memory vs Raw
+0.7 pp
-123.6K
-5.9K
-129.5K
cheaper
Workflow Memory substantially reduces token cost with nearly unchanged success.
Skill vs Raw
+5.5 pp
-29.8K
-4.4K
-34.2K
better and cheaper
Skill improves success while still reducing token use relative to Raw trajectories.
Skill vs Workflow Memory
+4.8 pp
+93.8K
+1.5K
+95.3K
better but costlier
Skill trades additional context for stronger execution performance.
Table 14: Complete Arm 1 embedding-retrieval results on SkillsBench. Entries report ranking metrics from Qwen3-Embedding-0.6B using task–skill-description similarity. Top-5 is omitted for k=5 because it covers the full candidate pool.
Pool
k
Top-1
Top-3
Top-5
P
P
R
F1
P
R
F1
Random
5
97.7
33.0
98.9
49.4
–
–
–
10
95.5
32.6
97.7
48.9
19.5
97.7
32.6
20
95.5
32.2
96.6
48.3
19.5
97.7
32.6
50
92.0
32.2
96.6
48.3
19.5
97.7
32.6
100
84.1
30.7
92.0
46.0
19.1
95.5
31.8
Similar
5
70.5
31.8
95.5
47.7
–
–
–
10
63.6
29.2
87.5
43.8
19.3
96.6
32.2
20
60.2
26.9
80.7
40.3
17.5
87.5
29.2
50
56.8
24.2
72.7
36.4
16.4
81.8
27.3
100
53.4
22.7
68.2
34.1
15.7
78.4
26.1
Dissimilar
5
96.6
33.0
98.9
49.4
–
–
–
10
96.6
32.2
96.6
48.3
19.8
98.9
33.0
20
96.6
32.2
96.6
48.3
19.5
97.7
32.6
50
94.3
32.2
96.6
48.3
19.5
97.7
32.6
100
93.2
32.2
96.6
48.3
19.3
96.6
32.2
Table 15: Complete Arm 2 and Arm 3 retrieval results on SkillsBench. Rows correspond to agent–model, distractor regime, and pool size. Arm 2 reports explicit-selection precision, recall, and F1; Arm 3 reports parsed actual-use precision, recall, F1, and downstream success. Dashes indicate excluded entries.
Agent / Model
Pool
k
Arm 2: Agent Selection
Arm 3: Real Execution
P
R
F1
P
R
F1
Succ.
Gemini CLI Gemini-3.1-Pro-Preview
Random
5
74.4
75.0
74.7
18.6
69.4
29.3
39.2
Random
10
72.7
72.7
72.7
7.1
66.4
12.9
38.1
Random
20
80.1
81.8
81.0
3.6
66.0
6.8
39.2
Random
50
81.2
84.1
82.6
1.4
65.8
2.8
37.6
Random
100
77.4
83.0
80.1
0.7
66.0
1.4
39.0
Similar
5
54.3
69.3
60.9
17.6
70.1
28.1
38.8
Similar
10
61.3
79.5
69.2
7.0
67.3
12.6
38.5
Similar
20
58.4
77.3
66.6
3.7
66.2
6.9
34.0
Similar
50
59.3
76.1
66.7
1.4
66.2
2.7
36.7
Similar
100
55.4
73.9
63.3
0.7
66.7
1.4
36.1
Dissimilar
5
74.4
76.1
75.3
14.6
63.2
23.7
34.0
Dissimilar
10
80.1
81.8
81.0
8.7
66.0
15.3
38.8
Dissimilar
20
82.4
83.0
82.7
5.1
65.5
9.5
35.6
Dissimilar
50
79.3
80.7
80.0
1.6
64.4
3.1
38.1
Dissimilar
100
83.5
84.1
83.8
0.7
61.6
1.5
34.5
Codex GPT-5.4
Random
5
81.8
84.1
82.9
33.1
41.6
36.8
24.3
Random
10
83.0
86.4
84.6
39.2
59.7
47.3
35.5
Random
20
84.1
87.5
85.8
35.3
69.3
46.8
40.9
Random
50
71.7
87.5
78.8
15.8
63.5
25.4
35.0
Random
100
62.2
85.2
71.9
8.1
73.6
14.6
44.8
Similar
5
51.9
81.8
63.5
51.3
72.4
60.0
44.5
Similar
10
44.4
83.0
57.8
37.5
66.0
47.9
40.7
Similar
20
35.8
77.3
48.9
27.7
66.1
39.1
44.3
Similar
50
37.8
76.1
50.5
13.1
61.8
21.6
42.3
Similar
100
31.9
70.5
43.9
6.7
54.3
11.9
43.0
Dissimilar
5
83.3
85.2
84.3
42.5
63.0
50.7
37.3
Dissimilar
10
83.0
85.2
84.1
29.7
61.3
40.0
35.0
Dissimilar
20
82.8
85.2
84.0
12.7
60.5
21.0
31.8
Dissimilar
50
77.5
85.2
81.2
7.2
65.1
12.9
39.5
Table 16: Complete numerical results for the outcome-annotation ablation. Entries report downstream success for skills constructed under the indicated trajectory mixture. normal exposes source-trajectory outcomes during construction; no-hint withholds them. Terminal-Bench-Pro entries use 130 trials per condition, with missing or infrastructure-error trials counted as failures.
Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: \textbf{When do skills help, why do they work, and where do they fail?} Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7\% of skill cases, versus 4.5\% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts. Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6\% to 3.3\%. Confusable distractors impair offline identification, yet downstream success remains stable; exact ground-truth invocation is neither sufficient nor necessary. Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation. These findings move evaluation beyond aggregate success rates and guide reliable self-evolving agents.
