긴 재무제표를 통째로 주고 공식 힌트를 빼면, 최신 AI들도 계산을 반쯤 틀린다

arXiv:2607.286612026-07-21

Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements

연구팀은 실제 상장기업 재무제표(최대 3만 2천 토큰 분량)를 그대로 사용해 LLM의 재무 계산 능력을 테스트하는 벤치마크 FinIndices를 만들었다. 단일 수치를 구하는 문제와 여러 지표를 표로 정리해야 하는 문제 두 가지로 나눠 시험한 결과, 공식을 알려주지 않으면 최상위 모델조차 정답률이 반토막 났다. 표 형식으로 여러 값을 한 번에 뽑아야 하는 과제에서는 모델들이 시간축을 헷갈리거나 회계 규칙을 대충 뭉개는 식으로 무너졌다.

METAL MEDIA 해설 도표

FinIndices 데이터 제작과 평가 흐름

증거 상태측정 결과가 보고됨

1. 재무제표 추출829개 상장기업의 원문 보고서에서 대차대조표·손익계산서·현금흐름표를 파싱하고 형식을 무작위로 바꾼다.
2. 문맥 구성·함정 삽입필요한 보고기간을 정렬하고, 불필요한 기간을 끼워넣거나 필수 자료를 빼는 함정을 심는다.
3. 문제·정답 생성 및 전문가 검증교과서 공식으로 프로그램이 정답을 계산하고, 전문가 3인의 다수결로 640개 최종 테스트 세트를 확정한다.
4. 힌트 유무 비교 평가Single-Index/Table-Index 두 형식으로 나눠, 공식 힌트가 있을 때와 없을 때 정답률 차이를 측정한다.
5. SFT를 통한 개선 시도Gemini가 생성한 추론 과정을 학습시켜 힌트 없는 조건에서의 성능 회복 정도를 확인한다.

METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

829개 상장기업, 384개 재무지표, 28개 보고기간에서 자동으로 문제-정답 쌍을 만들고 전문가 3인의 다수결 검증을 거쳐 68,307개 샘플의 FinIndices 벤치마크를 구축했다.
문제는 값 하나를 계산하는 Single-Index와 여러 지표를 표로 채우는 Table-Index 두 형식, 그리고 회계 지식·시간 처리·통계 기준 정합성 등 네 가지 능력 축으로 나뉜다.
일부러 필요 없는 기간을 끼워넣거나 필수 자료를 빼서 '정보 부족'이라고 답해야 하는 함정 문제도 포함했다.
공식 힌트를 주고 풀 때와 빼고 풀 때 성능 차이를 비교하고, 표 생성과 단일 값 계산의 난이도 차이를 비교했다.
증류된 추론 데이터로 모델을 추가 학습(SFT)시켜 힌트 없는 상황에서 성능이 얼마나 개선되는지도 측정했다.

Figure 1: Three canonical financial analysis workflows motivating FinIndices. Each case follows an input tables → structured reasoning module → tabular output pipeline: cross-entity valuation comparison (BYD / Tesla / Li Auto), longitudinal drift detection (Kweichow Moutai, 2019–2023), and cross-caliber macro reconciliation (China GDP under NBS / World Bank / IMF–PPP). Together they exemplify the three core challenges our benchmark targets: caliber unification, temporal-semantic disambiguation (stock/flow, mtd/ytd/ttm), and cross-indicator alignment with accurate composite computation. The bottom strip contrasts the failure modes of naive pipelines with the guarantees of structured table reasoning.

Table 1: Distribution of FinIndices along the capability axes. Queries are tagged by their dominant required skills, resulting in four mutually exclusive clusters. Proportions remain stable across all data splits.
Split	Query Paradigm	Domain	Caliber	Temporal + Caliber	Temporal + Domain	Total
#	%	#	%	#	%	#	%
Train	Single-Index	4,192	7.27	36,829	63.85	10,015	17.36	6,645	11.52	57,681
Table-Index	1,131	11.75	5,579	57.96	1,735	18.02	1,181	12.27	9,626
Subtotal	5,323	7.91	42,408	63.01	11,750	17.46	7,826	11.63	67,307
Dev	Single-Index	15	6.91	138	63.59	38	17.51	26	11.98	217
Table-Index	14	9.79	89	62.24	21	14.69	19	13.29	143
Subtotal	29	8.06	227	63.06	59	16.39	45	12.50	360
Test	Single-Index	34	7.04	311	64.39	82	16.98	56	11.59	483
Table-Index	22	14.01	86	54.78	32	20.38	17	10.83	157
Subtotal	56	8.75	397	62.03	114	17.81	73	11.41	640

Figure 2: Overview of the FinIndices data synthesis pipeline. The workflow consists of three stages: (1) Financial Statement Extraction parses and format-randomizes core tables from raw reports; (2) Context Construction & Index Alignment aligns temporal data and injects adversarial traps (e.g., missing info); and (3) Instruction & QA Pair Generation constructs prompts using canonical formulas and programmatic execution, followed by a rigorous Expert Validation phase employing majority voting to ensure absolute ground-truth fidelity.

Table 2: Context Length and Structural Complexity. FinIndices features an order-of-magnitude increase in context lengths compared to previous datasets. Furthermore, to resolve these queries, models must aggregate data across multiple interconnected reporting periods and, in Table-Index tasks, generate highly structured multi-value outputs.
Dataset	Samples	Context Length (Tokens)
Mean	Max	Min
FinQA (3)	8,281	1,003.8	3,250	168
TAT-QA (27)	14,883	590.7	4,140	113
FinIndices (Ours)	68,307	16,202.0	33,126	70
Temporal and Structural Complexity in FinIndices
Query Paradigm	Input Periods	Output Periods	Target Values Generated
Single-Index Queries	Avg: 1.95 (Max: 6)	Avg: 1.00 (Max: 1)	1.00 (Single value)
Table-Index Queries	Avg: 2.89 (Max: 9)	Avg: 2.03 (Max: 4)	Avg: 7.75 (Range: 2 – 20)

Table 3: Main Results on FinIndices. Models are categorized by weight availability, scale, and domain specificity. For iteratively evaluated general LLMs, Dev and Test results are averaged across 3 independent runs with standard deviations reported as subscripts. The end-to-end accuracy exposes a systemic performance collapse across both Single-Index and Table-Index tasks when explicit formula hints are removed. Top-2 Test results are highlighted in Red (1st) and Blue (2nd).
Models	With Hint (w/ Formula)		Without Hint (w/o Formula)
Single-Index	Table-Index		Single-Index	Table-Index
Train	Dev	Test	Train	Dev	Test		Train	Dev	Test	Train	Dev	Test
Closed-Source Models
Gemini-3.1-Pro (thinking)	66.29	69.05±0.87	79.61±1.05	34.50	26.57±1.51	70.70±0.30		55.95	58.96±1.21	64.90±0.82	17.35	9.09±0.33	38.22±0.52
Claude-Opus-4.8 (thinking)	66.91	72.66±0.78	78.46±0.57	31.46	23.08±0.99	65.61±1.38		56.65	60.62±1.32	65.79±0.46	16.51	9.79±0.99	38.85±1.38
GPT-5.5 (thinking)	65.69	74.46±1.04	77.29±0.34	32.90	22.84±0.66	66.88±0.52		54.42	59.17±0.13	60.91±1.10	15.12	11.66±0.33	34.18±0.30
GPT-5.4 (thinking)	65.88	71.58±0.89	75.79±0.81	33.38	25.17±0.75	65.61±0.92		55.96	59.02±0.78	57.04±0.84	14.77	10.49±0.41	31.21±0.62
Open-Weight Large Models ( ≥ 200B )
Qwen3.7-Max (thinking)	62.59	69.51±0.93	78.53±0.28	35.10	26.57±0.00	72.82±0.60		50.75	54.24±0.67	58.18±0.92	15.47	10.02±0.66	33.97±0.79
DeepSeek-V4-Pro (thinking)	64.75	66.00±1.23	69.20±0.79	25.28	15.62±1.74	48.20±1.31		52.85	50.69±0.32	52.74±0.38	9.86	6.29±0.57	22.72±2.10
GLM-5.2 (thinking)	51.14	57.41±0.68	60.50±1.09	16.77	11.19±0.99	36.94±0.52		43.17	45.05±0.13	48.82±1.12	7.27	5.59±0.57	19.53±1.83
GLM-5 (thinking)	51.28	50.93±0.65	50.85±0.74	16.84	12.59±0.52	35.03±0.68		41.22	35.21±0.48	36.97±0.55	7.98	5.59±0.28	19.11±0.42
Open-Weight Medium Models ( 50B – 200B )
Qwen3.7-Plus (thinking)	61.53	65.33±0.69	76.16±1.11	30.58	21.91±0.87	63.91±0.60		50.38	53.77±1.74	58.72±1.01	13.63	8.39±1.51	29.72±2.86
Qwen3.5-Plus (thinking)	61.56	60.70±0.75	68.78±0.88	29.64	26.57±0.55	63.69±0.72		48.91	54.19±0.68	50.11±0.74	12.99	8.39±0.35	26.75±0.65
Open-Weight Small Models ( < 50B )
General
Qwen3.5-35B-A3B-thinking	55.12	61.85±0.82	62.40±0.76	14.85	11.10±0.45	26.50±0.55		42.15	47.10±0.58	48.20±0.61	6.55	7.15±0.31	12.50±0.38
Domain-Specific Financial
DianJin-R1-32B (28)	37.94	45.97±0.81	41.29±0.75	5.01	2.90±0.18	11.41±0.42		29.40	34.00±0.66	32.39±0.58	2.08	0.74±0.05	5.33±0.21
Fin-o1-14B	24.94	22.89±0.45	28.16±0.53	1.12	0.71±0.08	2.67±0.15		18.60	20.87±0.38	18.32±0.41	0.55	0.00±0.00	1.97±0.11
XuanYuan-FinX1	9.15	6.82±0.25	12.45±0.33	1.26	0.70±0.05	1.91±0.10		5.85	7.12±0.20	5.33±0.18	0.51	0.00±0.00	0.64±0.04
Fin-R1 (13)	8.21	4.95±0.18	10.87±0.28	0.08	0.00±0.00	0.00±0.00		4.98	6.25±0.15	4.41±0.12	0.02	0.00±0.00	0.00±0.00
Llama-Fin-8B	7.92	4.50±0.12	10.15±0.22	0.00	0.00±0.00	0.00±0.00		4.65	5.81±0.11	4.12±0.09	0.00	0.00±0.00	0.00±0.00

Table 4: Improvements via Domain-Specific SFT. Our fine-tuned variant shows steady gains on general knowledge benchmarks and substantial improvements in complex numerical reasoning and tabular generation (Without Hint setting).
Models	Finance Knowledge	Finance Calc.	Finance Table Reasoning
CFinBench	FinEval	FLAME	In-House	FinMath	Single-Index	Table-Index
(17)	(8)	(7)			(w/o Hint)	(w/o Hint)
Ablation: Impact of Domain-Specific Fine-Tuning
Qwen3.5-35B-A3B-thinking (Base)	74.51	88.59	84.87	71.95	66.90	47.92	15.29
Ours (SFT on Base)	75.22	89.43	87.26	72.56	69.08	56.46	19.11
Absolute Improvement (Δ)	+0.72	+0.84	+2.39	+0.61	+2.19	+8.54	+3.82

Table 5: A Single-Index case testing deep Accounting Domain Understanding. To correctly compute Total Invested Capital, the model must avoid the trap of treating Notes Payable as interest-bearing debt, systematically classifying dozens of balance sheet items strictly according to financial analysis conventions.
Case 1: Single-Index Query for Total Invested Capital (Domain Understanding)
Company & Period	Company: 301607, Fute Technology Reporting period: 2024Y. Target values are computed using the ending balance of the 2024Y consolidated balance sheet.
Task Paradigm	Single-Index. The model must return one numerical value: Total Invested Capital (Equity Method). The final answer must be scaled to thousands and rounded to two decimal places.
Capability Axis	Domain Understanding. The query provides a high-level formula that subtracts “non-interest-bearing liabilities” from total capital. The model must systematically classify over 30 liability line items. Crucially, it must know that operating liabilities like Notes Payable are generally non-interest-bearing in Chinese accounting, whereas Lease Liabilities and Current Maturities of Non-Current Liabilities are interest-bearing.
Input Statements	(1) 2024Y Consolidated Balance Sheet (2) 2024Y Parent-Company Balance Sheet (Distractor)
User Instruction	“Calculate Fute Technology’s 2024Y Total Invested Capital (Equity Method), expressed in thousands and rounded to two decimal places.”
Injected Formula	∙ Total Invested Capital = Equity Attributable to Parent + Total Liabilities − Non-interest-bearing Current Liabilities − Non-interest-bearing Non-current Liabilities
Extracted Items	From the 2024Y Consolidated Balance Sheet: ∙ Equity Attributable to Parent: 1,000,020,938.49 ∙ Total Liabilities: 1,285,210,565.03 Interest-Bearing Liabilities (to be retained in Capital): ∙ Non-current Liab due within 1 year: 34,943,585.84 ∙ Long-term Borrowings: 48,296,660.07 ∙ Lease Liabilities: 16,566,591.38 ∙ Long-term Payables: 54,160,000.00 (Note: Notes Payable of 215,022,707.38 is correctly identified as an operating non-interest-bearing liability and must be deducted.)
Calculation Process	Step 1: Simplify the logic via Interest-Bearing Liabilities. Since Total Liab−Non-interest Liab=Interest-bearing Liab, the formula reduces to: Invested Capital=Equity Attributable to Parent+Interest-bearing Liab. Step 2: Sum the Interest-Bearing Liabilities. 34,943,585.84+48,296,660.07+16,566,591.38+54,160,000.00=153,966,837.29 Step 3: Add Equity to compute Total Invested Capital. 1,000,020,938.49+153,966,837.29=1,153,987,775.78 Step 4: Unit conversion and rounding. Divide by 1,000 (thousands): 1,153,987.77578⋯≈1,153,987.78
Ground Truth	1153987.78

Table 6: A Single-Index case requiring Temporal Reasoning and Domain Understanding. The model must navigate six non-sequential reporting periods, identify the correct three tables (Current, Previous Annual, Previous Same-Period), and compute a rolling Trailing Twelve Months (TTM) window while handling negative numbers and specific unit conversions.
Case 2: Single-Index Query for Trailing Twelve Months (TTM) Computation
Company & Period	Company: 301183, Dongtian Micro Target Period: TTM ending 2025H1 (2024-07-01 to 2025-06-30).
Task Paradigm	Single-Index. The model must return one numerical value: Operating Expenses (TTM). The final answer must be scaled to ten-thousands and rounded to two decimal places.
Capability Axis	Temporal Reasoning + Domain Understanding. The model must aggregate three specific expense lines (Selling, Administrative, and Financial) across non-sequential reporting periods using the TTM rolling-window logic: TTM=Current Period+Previous Annual−Previous Period.
Input Statements	The prompt provides six distinct financial reporting periods in a randomized, non-chronological order, mixing text tables and Python tuple lists: (1) 2025H1 Consolidated Income Statement (2) 2025Q1 Consolidated Income Statement (Distractor) (3) 2024Y (Annual) Consolidated Income Statement (4) 2024Q3 Consolidated Income Statement (Distractor) (5) 2024H1 Consolidated Income Statement (6) 2024Q1 Consolidated Income Statement (Distractor) In addition to consolidated statements, parent-company statements are provided as distractors.
User Instruction	“Calculate Dongtian Micro’s Operating Expenses (TTM) as of 2025-06-30. The TTM baseline date is the statement announcement date. Express the result in ten-thousands and round to two decimal places.”
Injected Formula	Operating Expenses (TTM)=Selling Expenses (TTM)+Administrative Expenses (TTM)+Financial Expenses (TTM) where TTM rule is defined as: ∙ Since the latest period (2025H1) is not an annual report, TTM=Current Period (2025H1)+Last Annual (2024Y)−Same Period Last Year (2024H1).
Extracted Items	From 2025H1 Consolidated Statement: ∙ Selling Expenses: 2,789,101.10 ∙ Administrative Expenses: 16,554,534.56 ∙ Financial Expenses: −2,457,530.39 From 2024Y Consolidated Statement: ∙ Selling Expenses: 4,958,105.99 ∙ Administrative Expenses: 34,543,848.50 ∙ Financial Expenses: −6,779,324.49 From 2024H1 Consolidated Statement: ∙ Selling Expenses: 2,236,714.86 ∙ Administrative Expenses: 12,031,506.74 ∙ Financial Expenses: −3,844,353.30
Calculation Process	Step 1: Compute TTM for each expense category. ∙ Selling Exp (TTM)=2,789,101.10+4,958,105.99−2,236,714.86=5,510,492.23 ∙ Admin Exp (TTM)=16,554,534.56+34,543,848.50−12,031,506.74=39,066,876.32 ∙ Financial Exp (TTM)=(−2,457,530.39)+(−6,779,324.49)−(−3,844,353.30)=−5,392,501.58 Step 2: Aggregate expenses and apply domain-specific netting adjustments. By substituting the extracted figures and aligning the expense caliber definitions as strictly expected by the ground truth logic, the model derives the net operating expense value. Step 3: Unit conversion and rounding. Convert the final aggregated value to ten-thousands (divide by 10,000) and round to two decimal places: ≈3,606.24
Ground Truth	3606.24

Table 7: An Adversarial Single-Index case testing Temporal Reasoning. The model is asked to compute a standalone quarterly growth rate but is only provided with Annual cumulative statements. A robust model must recognize the impossibility of temporal de-cumulation (Annual − Q3 = Q4) without the Q3 data, explicitly refusing to calculate rather than returning a flawed annual growth metric.
Case 3: Adversarial Single-Index Query for Standalone Quarter YoY Growth
Company & Period	Company: 301158, Deshi Shares Target Period: 2023Q4 Standalone Quarter (queried via the 2023Y Annual Report).
Task Paradigm	Single-Index (Adversarial). The model is asked to compute the YoY growth rate of a specific standalone quarter’s operating cash flow. Because critical temporal data (Q3 reports) is omitted from the prompt, the model must reject the calculation.
Capability Axis	Temporal Reasoning + Complex Caliber Alignment. To find a Q4 standalone flow variable (like Cash Flow), the model must subtract the Q3 cumulative (Year-to-Date) value from the Annual cumulative value. It must execute this temporal de-cumulation for both the current year and the prior year before applying the YoY caliber.
Input Statements	The prompt provides only the annual reports: (1) 2023Y (Annual) Consolidated Balance Sheet, Income Statement, Cash Flow (2) 2023Y (Annual) Parent-Company Statements (Distractors) Missing Critical Context: The 2023Q3 and 2022Q3 interim reports are deliberately excluded from the input.
User Instruction	“Calculate Deshi Shares’ 2023Q4 standalone Net Operating Cash Flow YoY Growth Rate. Express ratio data in percentages rounded to one decimal place.”
Injected Formula	∙ Standalone Quarter Net Operating Cash Flow YoY Growth = (Current Standalone Quarter Net Operating Cash Flow − Prior Year Same Standalone Quarter Net Operating Cash Flow) / ABS(Prior Year Same Standalone Quarter Net Operating Cash Flow) ×100%
Extracted Items	From the 2023Y Consolidated Cash Flow Statement: ∙ 2023Y (Cumulative Annual) Net Operating Cash Flow: 105,884,015.27 ∙ 2022Y (Cumulative Annual) Net Operating Cash Flow: 4,610,664.14 Missing Critical Items: ∙ 2023Q3 (First 9 Months) Net Operating Cash Flow: Not found ∙ 2022Q3 (First 9 Months) Net Operating Cash Flow: Not found
Calculation Process	Step 1: Identify the temporal alignment requirements. The query asks for the standalone quarter growth rate. Since the base period is the 2023Y report, the standalone quarter in question is Q4. Step 2: Attempt temporal de-cumulation. To get 2023Q4 standalone cash flow, the model must calculate: 2023Y Cumulative−2023Q3 Cumulative. Step 3: Recognize information deficiency. The model scans the provided tables and realizes the Q3 (first three quarters) reports are absent. Therefore, the standalone Q4 figures for both the current and prior years cannot be unrolled. Step 4: Reject calculation. The model correctly halts execution instead of erroneously substituting the annual YoY growth rate in place of the standalone quarterly YoY growth rate.
Ground Truth	Insufficient information to calculate the final value

Table 8: A Table-Index case requiring Complex Caliber Alignment. To correctly compute operating turnover metrics, the model must align point-in-time stock variables (Balance Sheet) with accumulated flow variables (Income Statement) by averaging beginning and ending balances. Furthermore, it must apply distinct scaling multipliers (90, 270, 360) depending on the semantic length of each specific column’s reporting period.
Case 4: Table-Index Query for Multi-Metric Caliber Alignment (Turnover & Operating Capabilities)
Company & Period	Company: 301197, Gongda Keya Target Periods: 2023Q3, 2023Y (Annual), and 2024Q1.
Task Paradigm	Table-Index. The model must generate a strictly formatted HTML table containing four derived operating metrics across three different reporting periods. The output must be rounded to one decimal place.
Capability Axis	Complex Caliber Alignment. The model must align flow variables (Income Statement metrics like Revenue) with stock variables (Balance Sheet metrics like Assets and Liabilities). This requires averaging the beginning and ending stock balances and correctly applying period-specific annualization multipliers (e.g., 90 days for Q1, 270 days for Q3, 360 days for Annual).
Input Statements	The prompt provides a massive dump of uncropped financial tables spanning three periods: (1) 2023Q3 Consolidated & Parent Balance Sheets, Income Statements, Cash Flows (2) 2023Y Consolidated & Parent Balance Sheets, Income Statements, Cash Flows (3) 2024Q1 Consolidated & Parent Balance Sheets, Income Statements, Cash Flows (Tens of thousands of tokens containing raw financial grids.)
User Instruction	“Extract or calculate the following four metrics for Gongda Keya across 2023Q3, 2023Y, and 2024Q1: Pre-receipts and Contract Liabilities Turnover Days, Cash Conversion Cycle, Net Asset Turnover, and AR & Contract Assets Turnover. Output the results as an HTML table wrapped in <table></table> tags, keeping one decimal place.”
Injected Formula	∙ Turnover Days = [360 for Annual, 90 for Q1, 180 for H1, 270 for Q3] / Turnover Rate ∙ Cash Conversion Cycle = Inventory Turnover Days + AR Turnover Days − AP Turnover Days ∙ Net Asset Turnover = Total Operating Revenue × 2 / (Beginning Net Assets + Ending Net Assets), where Net Assets = Total Assets − Total Liabilities ∙ AR & Contract Assets Turnover = Revenue × 2 / (Current AR + Current Contract Assets + Previous Year-end AR + Previous Year-end Contract Assets)
Calculation Process	Step 1: Metric extraction across periods. The model must extract Total Assets, Total Liabilities, Revenue, Inventory, AR, AP, Contract Assets, and Pre-receipts for the start and end of 2023Q3, 2023Y, and 2024Q1. Step 2: Caliber alignment (Averaging Stocks to match Flows). To compute Net Asset Turnover for 2024Q1, the model extracts 2024Q1 Revenue (flow) and divides it by the average of 2023Y Net Assets (beginning stock) and 2024Q1 Net Assets (ending stock). Step 3: Period-specific multipliers. When computing Turnover Days, the model must dynamically select the numerator: 270 for the 2023Q3 column, 360 for the 2023Y column, and 90 for the 2024Q1 column. Step 4: HTML Table Synthesis. Format the 12 resulting data points into a 2D HTML grid without generation collapse.
Ground Truth	<table> <tr><td>Metric</td><td>2023Q3</td><td>2023Y</td><td>2024Q1</td></tr> <tr><td>Pre-receipts & Contract Liab Turnover Days</td><td>51.7</td><td>39.1</td><td>75.6</td></tr> <tr><td>Cash Conversion Cycle</td><td>779.1</td><td>500.6</td><td>1030.5</td></tr> <tr><td>Net Asset Turnover</td><td>0.1</td><td>0.3</td><td>0.0</td></tr> <tr><td>AR & Contract Assets Turnover</td><td>0.4</td><td>0.8</td><td>0.1</td></tr> </table>

Table 9: A Table-Index case testing pure Domain Understanding. While computing Working Capital and Retained Earnings requires simple extraction and addition, computing Total Invested Capital demands an exhaustive, item-by-item classification of the liability structure. The model must apply this strict accounting taxonomy perfectly across four distinct reporting periods to successfully generate the matrix.
Case 5: Table-Index Query for Multi-Metric Domain Understanding (Capital & Equity Structuring)
Company & Period	Company: 300986, Zhite New Materials Target Periods: 2022Y, 2023Y, 2024Y, and 2025Q3.
Task Paradigm	Table-Index. The model must output a structured JSON tuple_list (a 2D array) containing three derived financial metrics across four different reporting periods. The output values must be scaled to ten-thousands and rounded to two decimal places.
Capability Axis	Domain Understanding. The primary bottleneck is computing the “Total Invested Capital”. The model is given a high-level formula that subtracts “non-interest-bearing liabilities” from total capital, meaning it must possess the deep accounting knowledge to systematically classify dozens of balance-sheet line items (e.g., separating interest-bearing debts like bonds and borrowings from non-interest-bearing operating payables like taxes, employee benefits, and contract liabilities).
Input Statements	A massive, uncropped sequence of financial tables: (1) 2022Y Consolidated & Parent Balance Sheets (2) 2023Y Consolidated & Parent Balance Sheets (3) 2024Y Consolidated & Parent Balance Sheets (4) 2025Q3 Consolidated Balance Sheets (Includes tens of thousands of tokens covering over 100 distinct line items per period.)
User Instruction	“Extract or calculate Zhite New Materials’ Working Capital, Retained Earnings, and Total Invested Capital for 2022Y, 2023Y, 2024Y, and 2025Q3. Unit: ten-thousands, rounded to 2 decimal places. Output as a JSON tuple list wrapped in ‘‘‘json tags.”
Injected Formula	∙ Working Capital = Total Current Assets − Total Current Liabilities ∙ Retained Earnings = Surplus Reserve + Unappropriated Profit ∙ Total Invested Capital = Total Shareholders’ Equity + Total Liabilities − Non-interest-bearing Current Liabilities − Non-interest-bearing Non-current Liabilities
Calculation Process	Step 1: Metric Extraction across 4 periods. The model retrieves the stated totals (e.g., Total Current Assets, Total Current Liabilities, Surplus Reserve, Unappropriated Profit) directly from the consolidated tables for each year/quarter. Step 2: Component Classification (The Domain Challenge). For the Total Invested Capital calculation, the model must scan the entire liabilities section for each period and deduct all non-interest-bearing items. It must correctly classify items such as Accounts Payable, Pre-receipts, Contract Liabilities, Employee Benefits Payable, and Taxes Payable as non-interest-bearing, leaving only items like Short/Long-term Borrowings, Bonds Payable, and Lease Liabilities in the capital base. Step 3: Scaling and JSON Structuring. The resulting arrays are divided by 10,000, rounded to two decimal places, and mapped into a strict 3-row by 4-column JSON array.
Ground Truth	‘‘‘json [ ["-19368.16", "20477.45", "-1423.26", "-27180.48"], ["75991.92", "68198.91", "73104.71", "82318.87"], ["274889.00", "363636.97", "401096.25", "422101.34"] ] ‘‘‘

Table 10: A Table-Index case testing both Temporal Reasoning and Caliber Alignment. The model is provided with accumulated (YTD) financial statements and must dynamically unroll them into discrete standalone quarters before executing cross-metric calculations (Gross Profit) and temporal comparisons (YoY Growth).
Case 6: Table-Index Query for Standalone Quarter YoY Growth (Temporal & Caliber Alignment)
Company & Period	Company: 300963, Zhongzhou Special Materials Target Periods: 2022Q1, 2022Q2 (Standalone), and 2022Q3 (Standalone).
Task Paradigm	Table-Index. The model must generate a strictly formatted JSON array (a 3x3 matrix) containing derived growth metrics. The output must be rounded to two decimal places (representing percentages, though output as raw string numbers per instruction).
Capability Axis	Temporal Reasoning + Complex Caliber Alignment. ∙ Temporal: The model must de-cumulate Year-to-Date (YTD) figures from H1 and Q3 reports to isolate standalone Q2 and Q3 figures for both the current year and the prior year. ∙ Caliber: It must compute “Gross Profit” by aligning standalone Revenue and standalone Cost before applying the Year-over-Year (YoY) growth formula.
Input Statements	A continuous time series of full financial reports: (1) 2021Y (Annual) Statements (2) 2022Q1 Statements (3) 2022H1 (First Half) Statements (4) 2022Q3 (First Three Quarters YTD) Statements (The prior-year baseline figures for 2021Q1, 2021H1, and 2021Q3 are embedded in the comparative columns of the 2022 reports.)
User Instruction	“Calculate the standalone quarterly YoY growth rates for Total Operating Revenue, Operating Profit, and Gross Profit for 2022Q1, 2022Q2, and 2022Q3. Express ratio data in percentages rounded to two decimal places. Output all final results as a JSON list wrapped in ‘‘‘json tags.”
Injected Formula	∙ Standalone YoY Growth = (Current Standalone Quarter − Prior Year Same Standalone Quarter) / ABS(Prior Year Same Standalone Quarter) ×100 ∙ Standalone Gross Profit = Standalone Operating Revenue − Standalone Operating Cost
Calculation Process	Step 1: Temporal De-cumulation (YTD to Standalone). To find 2022Q2 standalone revenue, the model must subtract 2022Q1 revenue from 2022H1 revenue. To find 2022Q3 standalone revenue, it must subtract 2022H1 revenue from 2022Q3 YTD revenue. This process must be repeated for Operating Profit, Operating Costs, and for the 2021 base-year figures. Step 2: Caliber Alignment (Gross Profit). Once the standalone Q1, Q2, and Q3 Revenues and Costs are unrolled, the model subtracts Cost from Revenue to derive the standalone Gross Profit for each quarter in both 2022 and 2021. Step 3: YoY Calculation and JSON Structuring. Apply the YoY formula to the derived standalone metrics. Format the resulting 9 values into a 3x3 JSON array.
Ground Truth	‘‘‘json [ ["11.27", "5.44", "52.17"], ["113.94", "75.23", "202.25"], ["32.07", "23.08", "58.49"] ] ‘‘‘

Table 11: Detailed Test Accuracy (%) on FinIndices (With Hint). Results are grouped by task paradigm and capability axes.
Capability Axis	Claude-4.5	Gemini-3.1	GPT-5.4	DS-V4-Pro	GLM-5	Qwen3.5-Plus	Qwen3.5-35B-A3B
Task: Single-Index
Domain Understanding	78.79	76.47	71.88	71.88	61.76	69.70	61.29
Caliber Alignment	64.00	84.90	81.44	67.91	54.13	72.92	67.43
Temporal + Caliber	59.42	74.67	68.12	59.26	46.25	69.57	61.54
Temporal + Domain	52.83	59.26	58.33	34.55	32.73	42.55	47.27
Task: Table-Index
Domain Understanding	63.64	63.64	72.73	68.18	45.45	68.18	27.27
Caliber Alignment	27.91	63.95	50.00	32.56	20.93	52.33	23.26
Temporal + Caliber	56.25	90.62	90.62	59.38	37.50	78.12	40.62
Temporal + Domain	76.47	76.47	88.24	76.47	88.24	88.24	29.41

Table 12: Detailed Test Accuracy (%) on FinIndices (Without Hint). The removal of explicit formulas causes a systemic collapse across all capability axes, particularly on pure domain and caliber alignment tasks.
Capability Axis	Claude-4.5	Gemini-3.1	GPT-5.4	DS-V4-Pro	GLM-5	Qwen3.5-Plus	Qwen3.5-35B-A3B
Task: Single-Index
Domain Understanding	55.88	62.50	48.48	42.42	37.50	43.75	48.48
Caliber Alignment	51.46	68.47	61.05	48.98	37.91	54.26	50.17
Temporal + Caliber	54.41	60.81	58.73	51.85	41.46	44.12	45.00
Temporal + Domain	54.72	51.92	38.78	35.85	25.00	40.00	50.00
Task: Table-Index
Domain Understanding	18.18	22.73	18.18	27.27	9.09	18.18	4.55
Caliber Alignment	12.79	23.26	15.12	11.63	9.30	13.95	9.30
Temporal + Caliber	37.50	68.75	59.38	50.00	25.00	56.25	15.62
Temporal + Domain	76.47	76.47	76.47	58.82	70.59	47.06	41.18

Table 13: Taxonomy of 5 representative failure modes for large-scale models. By contrasting flawed generation trajectories against correct domain logic, we expose severe vulnerabilities in temporal de-cumulation, caliber alignment, and formula adherence.
Case 1: Semantic Anchor Trap (Domain Axis)
Target & Context	Net Income from Value Changes \| 2025H1 Income Statement (RuiDi Drives).
Ground Truth	289.1 (10k RMB) \| Formula: Investment Income + Fair Value Change + Exchange Gain + Hedging Gain.
Erroneous Trajectory	-24.5. Triggered by the literal string “Value Change,” the model solely extracted “Fair value change income” (-245,236) and ignored the “Investment Income” (3,136,526) necessary for the full aggregated formula.
Root Cause	Semantic Rigidity. The model lacks the internal domain mapping to recognize that “Value Change” is an umbrella financial indicator, falling for a surface-level lexical match.
Case 2: Cognitive Overload in De-cumulation (Temporal Axis)
Target & Context	2022Q3 Single-Quarter Gross Profit YoY Growth \| 2022Q3 YTD, 2022H1, and 2021 statements (Zhongzhou).
Ground Truth	58.49% \| Formula: GPQ3_single=(RevYTD−RevH1)−(CostYTD−CostH1).
Erroneous Trajectory	202.25%. The model correctly computed the 2022Q3 single-quarter profit. However, for the baseline year (2021Q3), it lost track of the constraint and directly used the 2021Q3 YTD profit as the denominator.
Root Cause	Contextual Amnesia. When nesting temporal subtractions inside a comparative ratio formula, the attention mechanism fails to replicate the de-cumulation logic across historical reporting years.
Case 3: Ignoring Structural Caliber (Caliber Axis)
Target & Context	Annualized Return on Assets (ROA) \| 2025Q3 Income Statement, 2025Q3 & 2024Y Balance Sheets (Suneng).
Ground Truth	1.217% \| Formula: (NetProfitQ3×43)/AverageTotalAssets.
Erroneous Trajectory	1.208%. The model successfully annualized the profit flow but directly divided it by the Ending Total Assets (39.74B), completely ignoring the Beginning Assets.
Root Cause	Stock-Flow Mismatch. The model treats numerical extractions as flat variables, demonstrating a severe lack of caliber alignment awareness (averaging stocks to match flows) necessary for cross-statement integration.
Case 4: Rigid Formula Disobedience
Target & Context	Cash Return on Investment Ratio \| Xiangtan Elec. 2018Q3 (Round to 1 decimal).
Ground Truth	0.5 \| Formula: Net Operating CF / (Fixed Asset Cash Paid + Dividends - Financial Exp.)
Erroneous Trajectory	0.4. The model computed the denominator as 386,026,411.74.
Root Cause	Instruction Disobedience. The model failed to strictly follow the multi-step denominator formula provided in the prompt. It selectively omitted the required subtraction of “Financial Expenses”, showing rigidity when facing custom composite indicators.
Case 5: Conceptual Misalignment and Taxonomy Confusion
Target & Context	Tangible Asset Net Value \| Zhixin Precision 2025H1 (Unit: 10M RMB).
Ground Truth	97.0 \| Formula: Equity Attributable to Parent - (Intangibles + Goodwill + Deferred Tax Assets, etc.)
Erroneous Trajectory	99.8. The model used generic “Total Owner’s Equity” as the base.
Root Cause	Taxonomy Confusion. The model failed to distinguish the fine-grained boundary between “Total Equity” and “Equity Attributable to Parent Company”. Furthermore, it missed deductions required by new accounting standards (e.g., Deferred Tax Assets).

Table 14: Scaling collapse and format hallucination in small parameter models. The model fails the arithmetic scaling step (incorrect decimal shift) and hallucinates JSON dictionary keys instead of the requested nested tuple array.
Small Model Case: Scaling Collapse and Format Hallucination
Target Metric	Working Capital, Retained Earnings, Total Invested Capital (2022Y - 2025Q3)
Constraints	1. Unit: 10,000 RMB (Divide raw values by 104). 2. Format: tuple_list (e.g., [["Year", Val1, Val2, ...]]). 3. Precision: Round to 2 decimal places. Frequency: Annual basis.
Ground Truth	[["2022Y", -19368.16, 75991.92, 274889.0], ...]] (Strict division by 104, correct formula logic, compliant tuple formatting)
Model Output	[{"Year": "2022Y", "Working_Capital_10k": -1936.82, "Retained_Earnings_10k": 7599.19, "Total_Invested_Capital_10k": 27494.20}, ...}]

Table 15: Ablation of Temporal Misalignment. By evaluating identical YoY indicator types individually versus inside a unified HTML table, we isolate the failure trigger. Gemini possesses the temporal logic to fetch correct prior-period reports (evidenced by 100% isolated accuracy), but systematically substitutes wrong, adjacent columns (e.g., Beginning Balance) to lazily fulfill structural constraints under the cognitive load of multi-metric table generation.
Financial Indicator	Accounting & Calculation Logic	Isolated Query (single_indice)	Table Query (table_indice)	Bottleneck Analysis
Net Asset YoY Growth	Temporal mapping: Current End−Prior Year Same PeriodABS(Prior Year Same Period)	Xinlaifu: ✓ GT: 6.19% Pred: 6.19%	Tongdahai: ✗ GT: -4.22% Pred: -3.25%	Temporal Misalignment (Column Substitution). In isolation, the model correctly locates the Prior Year Same Period report. In the table, it lazily substitutes the adjacent “Beginning Balance” column to force-fill the structural schema: (1311.76M−1355.79M)/1355.79M=−3.25%.
Net Profit Cash Content YoY	Complex cross-statement derivation: Compute Operating CF / Net Profit, then calculate YoY.	Gelin Jingmi: ✓ GT: 12349.1% Pred: 12349.1%	Tongdahai: ✓ GT: -11.48% Pred: -11.48%	Maintained. The model can execute complex cross-statement math when the required YoY columns are explicitly aligned in the source statements.
Operating Profit YoY	Standard line-item YoY derivation: Deduct prior year operating profit; divide by ABS prior year.	Zhongzhou Tecai: ✓ GT: 6.0% Pred: 6.0%	Tongdahai: ✓ GT: -29.34% Pred: -29.34%	Maintained. Simple horizontal math extraction survives the structural load because the “Prior Period” column is explicitly provided in the Income Statement.

Table 16: Ablation of the Aggregation Shortcut. By evaluating the exact same five indicators individually versus as a unified table, we isolate the failure trigger. Gemini possesses the knowledge to solve complex operating adjustments (evidenced by 100% isolated accuracy), but systematically abandons this rigor for a shallow shortcut under the cognitive load of multi-metric table generation.
Financial Indicator	Accounting & Calculation Logic	Isolated Query (single_indice)	Table Query (table_indice)	Bottleneck Analysis
Operating Net / Total Profit	Deep CAS adjustment (incorporating Impairment Losses outside OpCost).	ST Tongde: ✓ GT: 104.0% Pred: 104.0%	Zhenyu Tech: ✗ GT: 94.06% Pred: 129.82%	Aggregation Shortcut. In isolation, Gemini executes the CAS logic perfectly. Inside the table, it defaults to a lazy literal (Rev−Cost)/Profit shortcut to save cognitive effort.
Single-Quarter ROE	Deduct prior quarter net profit; divide by average parent equity.	Dingrongyan: ✓ GT: 4.6% Pred: 4.6%	Zhenyu Tech: ✓ GT: 3.70% Pred: 3.70%	Maintained. Standard ratio logic survives the structural load.
Admin Expense / Revenue Ratio	Deduct prior quarter values to isolate single quarter; compute ratio.	Zhongjin: ✓ GT: 18.2% Pred: 18.2%	Zhenyu Tech: ✓ GT: 2.73% Pred: 2.73%	Maintained. Shallow numerator-denominator mapping remains stable.
Value-Change Net Income	Sum fair-value change, investment income, and FX income; adjust unit.	Zanyu Tech: ✓ GT: 13,333.95 Pred: 13,333.95	Zhenyu Tech: ✓ GT: 139,014.56 Pred: 139,014.56	Maintained. Additive aggregation remains unaffected by table constraints.
Investing Cash Flow Ratio	Aggregate three distinct CF categories; compute proportional share.	Tianxin: ✓ GT: -97.15% Pred: -97.15%	Zhenyu Tech: ✓ GT: -176.44% Pred: -176.44%	Maintained. Multi-step but structurally salient on the cash flow statement.

Table 17: Comprehensive Taxonomy of Formulaic Sources. The “Ratio” denotes the percentage of total formulas derived from each framework. Color coding: Target Ratio/Metric, Core Financial Item, and Adjustment/Normalization Factor.
Source Identifier	Ratio	Source Overview	Representative Examples
Financial Reporting Analysis (Revsine)	58.05%	An analytical framework utilized for global financial statement analysis. It establishes the methodology for period-to-period reconciliation and Trailing Twelve Months (TTM) adjustments to ensure cross-period comparability.	• OCF_Margin(TTM) = OCF(TTM) / Revenue(TTM) • ROIC(TTM) = NOPAT(TTM) / Avg_Invested_Capital • Net_Profit_Ratio(TTM) = Net_Profit(TTM) / Revenue(TTM)
Chinese Accounting Standards (CAS)	35.34%	The statutory reporting standard issued by the Ministry of Finance of China. It defines the rigorous line-item classification for listed companies across general and financial (Banking, Securities, Insurance) sectors.	• Interest_Bearing_Debt = Total_Liabs - Non-interest_Liabs • EBITDA = Op_Profit + Depr_Amort + Interest_Exp • Operating_Revenue = Net_Interest + Net_Fee + …
Textbooks	6.61%	Derived from authoritative global textbooks, including McKinsey’s Valuation, Ross’s Corporate Finance, and Kieso’s Intermediate Accounting, which serve as standard academic references.	• FCFF: EBIT(1-t) + D&A - CapEx - ΔWorking_Cap • Cash Cycle: Inv_Days + Rec_Days - Pay_Days • ROE: Net_Income / [(Equity_beg + Equity_end)/2]

실제로 확인된 결과

공식 힌트를 제거하자 Gemini-3.1-Pro의 표 작성 정답률이 70.70%(원문 요약 기준, 본문 상세치 70.07%)에서 38.22%로 떨어졌고, Claude-Opus-4.8도 38.85%에 그쳤다.
힌트가 있을 때도 Gemini-3.1-Pro는 단일 값 문제 79.61%에서 표 작성 문제 70.07%로 정답률이 낮아져, 여러 값을 표로 만드는 과제 자체가 더 어렵다는 것을 보였다.
재무 특화로 사전학습된 DianJin-R1-32B는 단일 값 문제에서는 41.29%를 기록했지만 표 작성 과제에서는 힌트가 있어도 11.41%, 없으면 5.33%로 완전히 무너졌고, Fin-R1과 Llama-Fin-8B 등 다른 재무 특화 모델은 거의 0%에 가까웠다.
힌트가 있는 표 작성 과제에서 통계 기준 정합성(caliber alignment)이 가장 취약해 최고 모델 Gemini-3.1도 63.95%, GPT-5.4는 50.00%에 머물렀고, 힌트를 빼면 GPT-5.4의 회계 이해 항목은 72.73%에서 18.18%로, Gemini의 정합성 항목은 63.95%에서 23.26%로 급락했다.
6,301건의 Gemini 증류 추론 데이터로 Qwen3.5-35B-A3B를 미세조정하자 힌트 없는 조건에서 단일 값 문제는 +8.54%, 표 작성 문제는 +3.82% 향상되었고, 기존 재무 지식 벤치마크(CFinBench, FinEval, FLAME 등)에서도 성능 저하 없이 +0.61%~+2.39% 개선됐다.

어디에 쓸 수 있나

재무 분석 AI 에이전트를 도입하기 전에 공식을 명시적으로 제공하지 않는 상태에서 얼마나 정확한지 별도로 검증하는 절차를 마련할 수 있다.
여러 재무지표를 한 번에 표로 뽑아내야 하는 자동화 파이프라인에서는 값 하나씩 검증하는 단계를 추가해 표 생성 단계의 오류를 걸러낼 수 있다.
재무 특화 모델을 도입할 때 단순 QA 성능뿐 아니라 다기간·다지표 표 생성 성능을 별도로 테스트해볼 근거로 쓸 수 있다.

한계와 남은 검증

평가는 중국 상장기업의 중국 회계기준 재무제표를 기반으로 하므로 다른 국가·회계기준에 그대로 적용되는지는 확인되지 않았다.
본문에 소개된 SFT 개선 효과는 특정 모델(Qwen3.5-35B-A3B)과 특정 증류 데이터 규모에서 측정된 것으로 다른 모델·데이터 조합에서도 재현되는지는 별도 검증이 필요하다.
벤치마크는 자동화된 정답 채점 방식(완전 일치 채점)을 쓰므로, 부분적으로 맞았지만 형식이 다른 답변이 과도하게 틀린 것으로 처리될 가능성이 있다.

왜 중요한가

재무 분석 업무는 결국 여러 표를 맞춰서 감사 가능한 최종 표를 만드는 일인데, 이 연구는 AI가 공식을 외웠을 뿐 실제로 이해하지는 못한다는 것을 구체적 수치로 보여준다. 재무 자동화나 AI 에이전트를 실무에 투입하려는 사람이라면 어떤 지점에서 신뢰가 깨지는지 미리 알아야 한다.

이 논문의 용어

Single-Index / Table-Index · 값 하나만 구하는 문제(Single) vs 여러 지표를 표 형식으로 채우는 문제(Table)
With Hint / Without Hint · 계산 공식을 프롬프트에 알려주는지 여부
TTM(Trailing Twelve Months) · 최근 12개월 실적을 여러 분기 자료를 합쳐 계산하는 방식
stock/flow 정합성 · 특정 시점의 잔액(재고 등)과 일정 기간 누적값(매출 등)을 맞춰 계산해야 하는 문제
SFT(지도 미세조정) · 정답이 포함된 추론 과정 데이터로 모델을 추가 학습시키는 방법

저자 · Xinke Tong

arXiv에서 원문 보기

METAL MEDIA 최신 기사

그림 출처: Xinke Tong et al., arXiv:2607.28661, arxiv-nonexclusive

긴 재무제표를 통째로 주고 공식 힌트를 빼면, 최신 AI들도 계산을 반쯤 틀린다

FinIndices 데이터 제작과 평가 흐름

무엇을 했나

실제로 확인된 결과

어디에 쓸 수 있나

한계와 남은 검증

왜 중요한가

이 논문의 용어

최신 논문

METAL MEDIA 최신 기사