Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
FinSkillBench: Evaluating AI Agents and Domain Skills for Investment Management
arXiv:2608.180992026-08-20
Giving AI investment agents a validated playbook boosts performance sharply, but letting them write their own doesn't help
FinSkillBench is a benchmark testing whether AI agents can handle real investment tasks across portfolio construction, risk management, and fundamental analysis, using 2,603 task episodes across 12 subtasks. Giving agents human-authored 'curated skill' packages containing validated procedures and executable code raised average scores from 0.366 to 0.528, while letting agents write their own 'self-generated' skills added cost with almost no benefit. The findings held up across 9 models in a 17,820-episode main run and were reproduced with a separate Hermes Agent framework across 8 models and 5,280 episodes.
METAL MEDIA explanatory visual
Giving AI investment agents a validated playbook boosts performance sharply, but letting them write their own doesn't help
01The benchmark covers 12 realistic investment tasks—portfolio optimization, mandate-violation monitoring, stress testing, financial-statement normalization, and more—each paired with an automated verifier that checks the agent's answer against hidden ground truth.
02Researchers held tasks, tools, and turn limits fixed while varying only what procedural resources agents could access: no skills at all, human-curated skill packages, or skills the agent writes for itself mid-task.
03Across 9 models and 17,820 evaluations, curated skills lifted the average score by 16.2 percentage points, with the biggest gains in portfolio construction and risk management tasks.
04Self-generated skills increased token usage and turn counts but only improved accuracy by 0.5 percentage points—essentially no measurable benefit.
05An independent replication using the Hermes Agent framework reproduced the same directional pattern across all three domains, confirming the result isn't specific to one agent implementation.
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
The benchmark covers 12 realistic investment tasks—portfolio optimization, mandate-violation monitoring, stress testing, financial-statement normalization, and more—each paired with an automated verifier that checks the agent's answer against hidden ground truth.
Researchers held tasks, tools, and turn limits fixed while varying only what procedural resources agents could access: no skills at all, human-curated skill packages, or skills the agent writes for itself mid-task.
Across 9 models and 17,820 evaluations, curated skills lifted the average score by 16.2 percentage points, with the biggest gains in portfolio construction and risk management tasks.
Self-generated skills increased token usage and turn counts but only improved accuracy by 0.5 percentage points—essentially no measurable benefit.
An independent replication using the Hermes Agent framework reproduced the same directional pattern across all three domains, confirming the result isn't specific to one agent implementation.
Table 1. Scorer-to-subtask mapping. Soft-vs-hard grading is calibrated to operational severity.
Table 2. Aggregate condition means, excluding Phi-4 (5,280 evaluations per condition). CIs are bootstrap 95% intervals (10,000 resamples) on paired deltas.
Condition
Mean
Δ vs. No-Skill
95% CI
No-Skill
0.366
—
—
Curated
0.528
+0.162
[+0.152, +0.171]
Self-Generated
0.371
+0.005
[−0.002, +0.011]
Table 3. Domain-level results (excl. Phi-4). PC and RM benefit most because curated skills contain validated solvers and precise input-passing instructions. CI shown for the curated − no-skill paired delta.
Table 7. Hermes harness: aggregate, by-domain, and by-model results. “†” indicates the curated and no-skill 95% CIs do not overlap. Numbers are unweighted means over 1,920 (overall, no-skill) and 5,280 (overall, curated skills) overall episodes. No-skill was a sample run of the first 20 episodes per each 12 subtask (240 total per model).
Group
N
No-Skill (95% CI)
N
Curated (95% CI)
Δ
Overall
All
1920
0.354 [0.338, 0.370]
5280
0.679 [0.669, 0.688]
+0.325
†
By domain
Fundamental analysis
480
0.439 [0.408, 0.470]
2400
0.567 [0.553, 0.582]
+0.129
†
Portfolio construction
800
0.325 [0.301, 0.350]
1600
0.760 [0.744, 0.776]
+0.435
†
Risk management
640
0.325 [0.297, 0.354]
1280
0.785 [0.767, 0.802]
+0.459
†
By model
claude-sonnet-4.6
240
0.304 [0.257, 0.353]
660
0.795 [0.775, 0.815]
+0.490
†
DeepSeek-V3.2
240
0.371 [0.325, 0.419]
660
0.686 [0.659, 0.712]
+0.315
†
gemini-2.5-pro
240
0.333 [0.291, 0.375]
660
0.477 [0.441, 0.512]
+0.144
†
gemma-4-31b-it
240
0.375 [0.332, 0.418]
660
0.642 [0.614, 0.670]
+0.267
†
glm-5.1
240
0.397 [0.350, 0.444]
660
0.779 [0.758, 0.801]
+0.383
†
gpt-4.1
240
0.285 [0.244, 0.327]
660
0.699 [0.672, 0.726]
+0.414
†
gpt-5.4
240
0.408 [0.365, 0.451]
660
0.660 [0.631, 0.687]
+0.252
†
grok-4.20
240
0.357 [0.314, 0.401]
660
0.691 [0.664, 0.719]
+0.334
†
Table 8. Tool availability by condition. “Always” means available in all three conditions; “Curated only” means available only when the curated skill package is mounted; “Self-gen only” means available only in the self-generated condition.
Tool
Availability
Description
submit_answer
Always
Submit final JSON answer (session-ending)
load_skill
Always
Load a SKILL.md into context (returns “not found” when empty)
load_references
Always
Load supplementary reference documents for a loaded skill
get_task_data
Always
Retrieve specific fields from the task input at full precision
query_xbrl
Always (FA)
Query the XBRL financial data panel by ticker, period, and metrics
run_skill_ script
Curated only
Execute a validated Python script from the skill directory
save_skill
Self-gen only
Write a SKILL.md to the per-episode scratch directory
Table 9. Cognitive and procedural demands by subtask. ✓= primary demand; ∘ = secondary demand.
Subtask
Numerical optim.
Tool param.
Multi-step comp.
Domain knowledge
Data retrieval
Precision mgmt.
Multi-output
Unconstrained optim.
✓
✓
∘
∘
∘
✓
Constrained optim.
✓
✓
∘
✓
∘
✓
✓
Tool-use param.
✓
✓
Rebalancing
✓
✓
✓
∘
∘
✓
✓
Black–Litterman
✓
✓
✓
✓
∘
✓
✓
Constraint monitoring
✓
✓
∘
✓
Risk identification
✓
✓
✓
✓
Stress testing
∘
✓
✓
✓
✓
✓
Risk remediation
✓
∘
✓
✓
∘
✓
Normalization
∘
✓
✓
Earnings quality
✓
✓
✓
✓
Driver decomposition
✓
✓
✓
✓
Table 10. Complete scorer specifications. Most component scores are clipped or averaged into [0,1]; the exp05 ranked-list scorer does not final-clip NDCG or the composite and can slightly exceed 1 when duplicate predicted risk types receive repeated relevance credit.
Scorer
Subtask
Formula
L2 distance
Unconstrained optim.
max(0,min(1, 1−∥wp−we∥2/(4θ))), θ=0.05
Constraint gate + L2
Constrained optim.
All expected constraint_satisfaction keys must be present and equal; extra predicted keys are ignored. If the gate passes, weight L2 score as above; else 0
Parameter match
Tool-use param.
Fraction of union leaf parameters matching (numeric: relative error < 0.2; categorical: exact; extra/missing leaves count against the denominator)
Turnover compliance
Rebalancing
0.5⋅sL2+0.25⋅sturn+0.25⋅strade, where sL2=max(0,min(1,1−L2/0.2)), sturn=max(0,1−min(1,|T^−T|/max(|T|,10−9))), and strade is mean expected-ticker trade consistency
View + weights
Black–Litterman
Mean of active components: posterior-returns MAE score max(0, 1−MAE/0.05) and weight L2 score max(0,min(1,1−L2/(4θ))) with θ=0.10
Exact match
Constraint monitoring
0.4⋅ 1[compliant match]+0.6⋅constraint accuracy, where accuracy is status match over expected normalized constraint types
Ranked recall
Risk identification
0.4⋅recall@k+0.3⋅precision@k+0.3⋅NDCG@k; recall/precision use multiset type overlap; exp05 NDCG uses max magnitude per expected type and is not final-clipped
Fraction of scored non-metadata metrics with relative error ≤ per-metric tolerance (default 5%); unwraps metrics wrapper when present on both sides
EQ composite
Earnings quality
Weighted mean over active terms: component accuracy (episode weight 0.4), Beneish-flag match (0.3), flag F1 (0.3); numeric-ratio accuracy is included only if numeric_ratio_weight>0 in exp05/zqbok_experiment05
Driver F1
Driver decomposition
Mean of active terms: revenue-driver recall/precision, matched revenue-driver direction, matched revenue numeric accuracy, margin recall/direction, matched margin numeric accuracy, and revenue-Δ accuracy
Why it matters
This shows that for deploying AI agents in real investment workflows, giving them reliable, validated procedures and tools may matter as much as which underlying model is used. It also cautions that letting agents invent their own procedures on the fly is not yet a trustworthy substitute.
Terms in this paper
agentic AI · AI systems that go beyond generating text to call tools and carry out multi-step tasks
skill · a reusable package of procedural documents and executable code an agent can load and use
curated skill · a human-authored, validated package of instructions and scripts
self-generated skill · a skill the agent writes for itself during a task before using it to answer
point-in-time data · data restricted to what was knowable as of a specific date, preventing lookahead information
Original abstract (English)
Investment management is a high-stakes domain in which agentic AI systems must do more than generate plausible text. They must retrieve point-in-time data, assemble correct computational inputs, invoke specialized methods, and produce auditable structured outputs. We introduce FinSkillBench, an evaluation suite designed to measure whether language model agents can effectively use financial domain skills to solve investment management tasks. The benchmark spans three domains, portfolio construction, risk management, and fundamental analysis, and includes 12 subtasks with 2,603 task episodes.
Each episode provides point-in-time inputs, hidden ground truth, and a task-specific verifier.We compare three conditions: no skill, curated skill packages consisting of procedural documents and executable components, and self-generated skills in which the agent writes and reuses its own procedures within an episode. Across 9 models and a large-scale evaluation, curated skills consistently improve performance, raising mean scores from 0.366 to 0.528, with the largest gains in portfolio construction and risk management.
In contrast, self-generated skills provide little benefit despite higher computational cost. An independent evaluation using a separate agent framework (Hermes Agent, 8 models, 5,280 episodes total) reproduces the directional pattern across all three domains, with the magnitude of skill effects varying by subtask and harness.
These results showthat in investment management agents, access to reliable procedural skills can be as important as model choice, while naive self-generation of skills is often ineffective. We release the benchmark, evaluation tools, curated skill packages, and full trajectories to support further research.
