Table 2. Aggregate condition means, excluding Phi-4 (5,280 evaluations per condition). CIs are bootstrap 95% intervals (10,000 resamples) on paired deltas.
Condition
Mean
Δ vs. No-Skill
95% CI
No-Skill
0.366
—
—
Curated
0.528
+0.162
[+0.152, +0.171]
Self-Generated
0.371
+0.005
[−0.002, +0.011]
Table 3. Domain-level results (excl. Phi-4). PC and RM benefit most because curated skills contain validated solvers and precise input-passing instructions. CI shown for the curated − no-skill paired delta.
Table 7. Hermes harness: aggregate, by-domain, and by-model results. “†” indicates the curated and no-skill 95% CIs do not overlap. Numbers are unweighted means over 1,920 (overall, no-skill) and 5,280 (overall, curated skills) overall episodes. No-skill was a sample run of the first 20 episodes per each 12 subtask (240 total per model).
Group
N
No-Skill (95% CI)
N
Curated (95% CI)
Δ
Overall
All
1920
0.354 [0.338, 0.370]
5280
0.679 [0.669, 0.688]
+0.325
†
By domain
Fundamental analysis
480
0.439 [0.408, 0.470]
2400
0.567 [0.553, 0.582]
+0.129
†
Portfolio construction
800
0.325 [0.301, 0.350]
1600
0.760 [0.744, 0.776]
+0.435
†
Risk management
640
0.325 [0.297, 0.354]
1280
0.785 [0.767, 0.802]
+0.459
†
By model
claude-sonnet-4.6
240
0.304 [0.257, 0.353]
660
0.795 [0.775, 0.815]
+0.490
†
DeepSeek-V3.2
240
0.371 [0.325, 0.419]
660
0.686 [0.659, 0.712]
+0.315
†
gemini-2.5-pro
240
0.333 [0.291, 0.375]
660
0.477 [0.441, 0.512]
+0.144
†
gemma-4-31b-it
240
0.375 [0.332, 0.418]
660
0.642 [0.614, 0.670]
+0.267
†
glm-5.1
240
0.397 [0.350, 0.444]
660
0.779 [0.758, 0.801]
+0.383
†
gpt-4.1
240
0.285 [0.244, 0.327]
660
0.699 [0.672, 0.726]
+0.414
†
gpt-5.4
240
0.408 [0.365, 0.451]
660
0.660 [0.631, 0.687]
+0.252
†
grok-4.20
240
0.357 [0.314, 0.401]
660
0.691 [0.664, 0.719]
+0.334
†
Table 8. Tool availability by condition. “Always” means available in all three conditions; “Curated only” means available only when the curated skill package is mounted; “Self-gen only” means available only in the self-generated condition.
Tool
Availability
Description
submit_answer
Always
Submit final JSON answer (session-ending)
load_skill
Always
Load a SKILL.md into context (returns “not found” when empty)
load_references
Always
Load supplementary reference documents for a loaded skill
get_task_data
Always
Retrieve specific fields from the task input at full precision
query_xbrl
Always (FA)
Query the XBRL financial data panel by ticker, period, and metrics
run_skill_ script
Curated only
Execute a validated Python script from the skill directory
save_skill
Self-gen only
Write a SKILL.md to the per-episode scratch directory
Table 9. Cognitive and procedural demands by subtask. ✓= primary demand; ∘ = secondary demand.
Subtask
Numerical optim.
Tool param.
Multi-step comp.
Domain knowledge
Data retrieval
Precision mgmt.
Multi-output
Unconstrained optim.
✓
✓
∘
∘
∘
✓
Constrained optim.
✓
✓
∘
✓
∘
✓
✓
Tool-use param.
✓
✓
Rebalancing
✓
✓
✓
∘
∘
✓
✓
Black–Litterman
✓
✓
✓
✓
∘
✓
✓
Constraint monitoring
✓
✓
∘
✓
Risk identification
✓
✓
✓
✓
Stress testing
∘
✓
✓
✓
✓
✓
Risk remediation
✓
∘
✓
✓
∘
✓
Normalization
∘
✓
✓
Earnings quality
✓
✓
✓
✓
Driver decomposition
✓
✓
✓
✓
Table 10. Complete scorer specifications. Most component scores are clipped or averaged into [0,1]; the exp05 ranked-list scorer does not final-clip NDCG or the composite and can slightly exceed 1 when duplicate predicted risk types receive repeated relevance credit.
Scorer
Subtask
Formula
L2 distance
Unconstrained optim.
max(0,min(1, 1−∥wp−we∥2/(4θ))), θ=0.05
Constraint gate + L2
Constrained optim.
All expected constraint_satisfaction keys must be present and equal; extra predicted keys are ignored. If the gate passes, weight L2 score as above; else 0
Parameter match
Tool-use param.
Fraction of union leaf parameters matching (numeric: relative error < 0.2; categorical: exact; extra/missing leaves count against the denominator)
Turnover compliance
Rebalancing
0.5⋅sL2+0.25⋅sturn+0.25⋅strade, where sL2=max(0,min(1,1−L2/0.2)), sturn=max(0,1−min(1,|T^−T|/max(|T|,10−9))), and strade is mean expected-ticker trade consistency
View + weights
Black–Litterman
Mean of active components: posterior-returns MAE score max(0, 1−MAE/0.05) and weight L2 score max(0,min(1,1−L2/(4θ))) with θ=0.10
Exact match
Constraint monitoring
0.4⋅ 1[compliant match]+0.6⋅constraint accuracy, where accuracy is status match over expected normalized constraint types
Ranked recall
Risk identification
0.4⋅recall@k+0.3⋅precision@k+0.3⋅NDCG@k; recall/precision use multiset type overlap; exp05 NDCG uses max magnitude per expected type and is not final-clipped
Fraction of scored non-metadata metrics with relative error ≤ per-metric tolerance (default 5%); unwraps metrics wrapper when present on both sides
EQ composite
Earnings quality
Weighted mean over active terms: component accuracy (episode weight 0.4), Beneish-flag match (0.3), flag F1 (0.3); numeric-ratio accuracy is included only if numeric_ratio_weight>0 in exp05/zqbok_experiment05
Driver F1
Driver decomposition
Mean of active terms: revenue-driver recall/precision, matched revenue-driver direction, matched revenue numeric accuracy, margin recall/direction, matched margin numeric accuracy, and revenue-Δ accuracy
Investment management is a high-stakes domain in which agentic AI systems must do more than generate plausible text. They must retrieve point-in-time data, assemble correct computational inputs, invoke specialized methods, and produce auditable structured outputs. We introduce FinSkillBench, an evaluation suite designed to measure whether language model agents can effectively use financial domain skills to solve investment management tasks. The benchmark spans three domains, portfolio construction, risk management, and fundamental analysis, and includes 12 subtasks with 2,603 task episodes.
Each episode provides point-in-time inputs, hidden ground truth, and a task-specific verifier.We compare three conditions: no skill, curated skill packages consisting of procedural documents and executable components, and self-generated skills in which the agent writes and reuses its own procedures within an episode. Across 9 models and a large-scale evaluation, curated skills consistently improve performance, raising mean scores from 0.366 to 0.528, with the largest gains in portfolio construction and risk management.
In contrast, self-generated skills provide little benefit despite higher computational cost. An independent evaluation using a separate agent framework (Hermes Agent, 8 models, 5,280 episodes total) reproduces the directional pattern across all three domains, with the magnitude of skill effects varying by subtask and harness.
These results showthat in investment management agents, access to reliable procedural skills can be as important as model choice, while naive self-generation of skills is often ineffective. We release the benchmark, evaluation tools, curated skill packages, and full trajectories to support further research.
