Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements

arXiv:2607.286612026-07-21

Feed AI the full, uncropped financial statements and take away the formula hints, and even top models get numbers wrong half the time

Researchers built FinIndices, a benchmark that tests LLMs on real, uncropped corporate financial statements (up to 32K tokens) instead of cropped table snippets. Models are tested on both single-number calculations and multi-metric table generation, and when explicit formulas are removed, even leading models' accuracy collapses dramatically. On table-generation tasks especially, models lose track of time periods or take accounting shortcuts that break correctness.

METAL MEDIA explanatory visual

FinIndices construction and evaluation pipeline

Evidence statusMeasured results reported

1. Statement extractionParses balance sheets, income statements, and cash flow statements from 829 companies' raw reports and randomizes their format.
2. Context building & trap injectionAligns the exact reporting periods needed and injects distractor periods or deliberately omits required data.
3. QA generation & expert validationProgrammatically computes ground-truth answers from textbook formulas, then three experts vote to finalize a 640-instance test set.
4. Hint on/off comparisonEvaluates Single-Index and Table-Index tasks both with and without explicit formula hints to measure the accuracy gap.
5. SFT recovery attemptFine-tunes a model on Gemini-distilled reasoning trajectories and measures how much hintless accuracy is regained.

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

The team built FinIndices, an automatically synthesized and expert-validated benchmark of 68,307 samples drawn from 829 listed companies, 384 financial indices, and 28 reporting periods.
Queries fall into two formats — Single-Index (compute one number) and Table-Index (fill in a multi-metric table) — spanning four capability axes including accounting-domain knowledge, temporal reasoning, and caliber (stock vs. flow) alignment.
The dataset injects adversarial traps: irrelevant extra periods to distract the model, and missing required periods that should force the model to answer 'insufficient information' instead of guessing.
Models were compared under 'With Hint' (formula given) and 'Without Hint' (formula withheld) conditions, and Single-Index vs. Table-Index task difficulty was also compared.
The team also fine-tuned a model on distilled reasoning trajectories to see how much of the lost accuracy under the hintless condition could be recovered.

Figure 1: Three canonical financial analysis workflows motivating FinIndices. Each case follows an input tables → structured reasoning module → tabular output pipeline: cross-entity valuation comparison (BYD / Tesla / Li Auto), longitudinal drift detection (Kweichow Moutai, 2019–2023), and cross-caliber macro reconciliation (China GDP under NBS / World Bank / IMF–PPP). Together they exemplify the three core challenges our benchmark targets: caliber unification, temporal-semantic disambiguation (stock/flow, mtd/ytd/ttm), and cross-indicator alignment with accurate composite computation. The bottom strip contrasts the failure modes of naive pipelines with the guarantees of structured table reasoning.

Table 1: Distribution of FinIndices along the capability axes. Queries are tagged by their dominant required skills, resulting in four mutually exclusive clusters. Proportions remain stable across all data splits.
Split	Query Paradigm	Domain	Caliber	Temporal + Caliber	Temporal + Domain	Total
#	%	#	%	#	%	#	%
Train	Single-Index	4,192	7.27	36,829	63.85	10,015	17.36	6,645	11.52	57,681
Table-Index	1,131	11.75	5,579	57.96	1,735	18.02	1,181	12.27	9,626
Subtotal	5,323	7.91	42,408	63.01	11,750	17.46	7,826	11.63	67,307
Dev	Single-Index	15	6.91	138	63.59	38	17.51	26	11.98	217
Table-Index	14	9.79	89	62.24	21	14.69	19	13.29	143
Subtotal	29	8.06	227	63.06	59	16.39	45	12.50	360
Test	Single-Index	34	7.04	311	64.39	82	16.98	56	11.59	483
Table-Index	22	14.01	86	54.78	32	20.38	17	10.83	157
Subtotal	56	8.75	397	62.03	114	17.81	73	11.41	640

Figure 2: Overview of the FinIndices data synthesis pipeline. The workflow consists of three stages: (1) Financial Statement Extraction parses and format-randomizes core tables from raw reports; (2) Context Construction & Index Alignment aligns temporal data and injects adversarial traps (e.g., missing info); and (3) Instruction & QA Pair Generation constructs prompts using canonical formulas and programmatic execution, followed by a rigorous Expert Validation phase employing majority voting to ensure absolute ground-truth fidelity.

Table 2: Context Length and Structural Complexity. FinIndices features an order-of-magnitude increase in context lengths compared to previous datasets. Furthermore, to resolve these queries, models must aggregate data across multiple interconnected reporting periods and, in Table-Index tasks, generate highly structured multi-value outputs.
Dataset	Samples	Context Length (Tokens)
Mean	Max	Min
FinQA (3)	8,281	1,003.8	3,250	168
TAT-QA (27)	14,883	590.7	4,140	113
FinIndices (Ours)	68,307	16,202.0	33,126	70
Temporal and Structural Complexity in FinIndices
Query Paradigm	Input Periods	Output Periods	Target Values Generated
Single-Index Queries	Avg: 1.95 (Max: 6)	Avg: 1.00 (Max: 1)	1.00 (Single value)
Table-Index Queries	Avg: 2.89 (Max: 9)	Avg: 2.03 (Max: 4)	Avg: 7.75 (Range: 2 – 20)

Table 3: Main Results on FinIndices. Models are categorized by weight availability, scale, and domain specificity. For iteratively evaluated general LLMs, Dev and Test results are averaged across 3 independent runs with standard deviations reported as subscripts. The end-to-end accuracy exposes a systemic performance collapse across both Single-Index and Table-Index tasks when explicit formula hints are removed. Top-2 Test results are highlighted in Red (1st) and Blue (2nd).
Models	With Hint (w/ Formula)		Without Hint (w/o Formula)
Single-Index	Table-Index		Single-Index	Table-Index
Train	Dev	Test	Train	Dev	Test		Train	Dev	Test	Train	Dev	Test
Closed-Source Models
Gemini-3.1-Pro (thinking)	66.29	69.05±0.87	79.61±1.05	34.50	26.57±1.51	70.70±0.30		55.95	58.96±1.21	64.90±0.82	17.35	9.09±0.33	38.22±0.52
Claude-Opus-4.8 (thinking)	66.91	72.66±0.78	78.46±0.57	31.46	23.08±0.99	65.61±1.38		56.65	60.62±1.32	65.79±0.46	16.51	9.79±0.99	38.85±1.38
GPT-5.5 (thinking)	65.69	74.46±1.04	77.29±0.34	32.90	22.84±0.66	66.88±0.52		54.42	59.17±0.13	60.91±1.10	15.12	11.66±0.33	34.18±0.30
GPT-5.4 (thinking)	65.88	71.58±0.89	75.79±0.81	33.38	25.17±0.75	65.61±0.92		55.96	59.02±0.78	57.04±0.84	14.77	10.49±0.41	31.21±0.62
Open-Weight Large Models ( ≥ 200B )
Qwen3.7-Max (thinking)	62.59	69.51±0.93	78.53±0.28	35.10	26.57±0.00	72.82±0.60		50.75	54.24±0.67	58.18±0.92	15.47	10.02±0.66	33.97±0.79
DeepSeek-V4-Pro (thinking)	64.75	66.00±1.23	69.20±0.79	25.28	15.62±1.74	48.20±1.31		52.85	50.69±0.32	52.74±0.38	9.86	6.29±0.57	22.72±2.10
GLM-5.2 (thinking)	51.14	57.41±0.68	60.50±1.09	16.77	11.19±0.99	36.94±0.52		43.17	45.05±0.13	48.82±1.12	7.27	5.59±0.57	19.53±1.83
GLM-5 (thinking)	51.28	50.93±0.65	50.85±0.74	16.84	12.59±0.52	35.03±0.68		41.22	35.21±0.48	36.97±0.55	7.98	5.59±0.28	19.11±0.42
Open-Weight Medium Models ( 50B – 200B )
Qwen3.7-Plus (thinking)	61.53	65.33±0.69	76.16±1.11	30.58	21.91±0.87	63.91±0.60		50.38	53.77±1.74	58.72±1.01	13.63	8.39±1.51	29.72±2.86
Qwen3.5-Plus (thinking)	61.56	60.70±0.75	68.78±0.88	29.64	26.57±0.55	63.69±0.72		48.91	54.19±0.68	50.11±0.74	12.99	8.39±0.35	26.75±0.65
Open-Weight Small Models ( < 50B )
General
Qwen3.5-35B-A3B-thinking	55.12	61.85±0.82	62.40±0.76	14.85	11.10±0.45	26.50±0.55		42.15	47.10±0.58	48.20±0.61	6.55	7.15±0.31	12.50±0.38
Domain-Specific Financial
DianJin-R1-32B (28)	37.94	45.97±0.81	41.29±0.75	5.01	2.90±0.18	11.41±0.42		29.40	34.00±0.66	32.39±0.58	2.08	0.74±0.05	5.33±0.21
Fin-o1-14B	24.94	22.89±0.45	28.16±0.53	1.12	0.71±0.08	2.67±0.15		18.60	20.87±0.38	18.32±0.41	0.55	0.00±0.00	1.97±0.11
XuanYuan-FinX1	9.15	6.82±0.25	12.45±0.33	1.26	0.70±0.05	1.91±0.10		5.85	7.12±0.20	5.33±0.18	0.51	0.00±0.00	0.64±0.04
Fin-R1 (13)	8.21	4.95±0.18	10.87±0.28	0.08	0.00±0.00	0.00±0.00		4.98	6.25±0.15	4.41±0.12	0.02	0.00±0.00	0.00±0.00
Llama-Fin-8B	7.92	4.50±0.12	10.15±0.22	0.00	0.00±0.00	0.00±0.00		4.65	5.81±0.11	4.12±0.09	0.00	0.00±0.00	0.00±0.00

Table 4: Improvements via Domain-Specific SFT. Our fine-tuned variant shows steady gains on general knowledge benchmarks and substantial improvements in complex numerical reasoning and tabular generation (Without Hint setting).
Models	Finance Knowledge	Finance Calc.	Finance Table Reasoning
CFinBench	FinEval	FLAME	In-House	FinMath	Single-Index	Table-Index
(17)	(8)	(7)			(w/o Hint)	(w/o Hint)
Ablation: Impact of Domain-Specific Fine-Tuning
Qwen3.5-35B-A3B-thinking (Base)	74.51	88.59	84.87	71.95	66.90	47.92	15.29
Ours (SFT on Base)	75.22	89.43	87.26	72.56	69.08	56.46	19.11
Absolute Improvement (Δ)	+0.72	+0.84	+2.39	+0.61	+2.19	+8.54	+3.82

Table 5: A Single-Index case testing deep Accounting Domain Understanding. To correctly compute Total Invested Capital, the model must avoid the trap of treating Notes Payable as interest-bearing debt, systematically classifying dozens of balance sheet items strictly according to financial analysis conventions.
Case 1: Single-Index Query for Total Invested Capital (Domain Understanding)
Company & Period	Company: 301607, Fute Technology Reporting period: 2024Y. Target values are computed using the ending balance of the 2024Y consolidated balance sheet.
Task Paradigm	Single-Index. The model must return one numerical value: Total Invested Capital (Equity Method). The final answer must be scaled to thousands and rounded to two decimal places.
Capability Axis	Domain Understanding. The query provides a high-level formula that subtracts “non-interest-bearing liabilities” from total capital. The model must systematically classify over 30 liability line items. Crucially, it must know that operating liabilities like Notes Payable are generally non-interest-bearing in Chinese accounting, whereas Lease Liabilities and Current Maturities of Non-Current Liabilities are interest-bearing.
Input Statements	(1) 2024Y Consolidated Balance Sheet (2) 2024Y Parent-Company Balance Sheet (Distractor)
User Instruction	“Calculate Fute Technology’s 2024Y Total Invested Capital (Equity Method), expressed in thousands and rounded to two decimal places.”
Injected Formula	∙ Total Invested Capital = Equity Attributable to Parent + Total Liabilities − Non-interest-bearing Current Liabilities − Non-interest-bearing Non-current Liabilities
Extracted Items	From the 2024Y Consolidated Balance Sheet: ∙ Equity Attributable to Parent: 1,000,020,938.49 ∙ Total Liabilities: 1,285,210,565.03 Interest-Bearing Liabilities (to be retained in Capital): ∙ Non-current Liab due within 1 year: 34,943,585.84 ∙ Long-term Borrowings: 48,296,660.07 ∙ Lease Liabilities: 16,566,591.38 ∙ Long-term Payables: 54,160,000.00 (Note: Notes Payable of 215,022,707.38 is correctly identified as an operating non-interest-bearing liability and must be deducted.)
Calculation Process	Step 1: Simplify the logic via Interest-Bearing Liabilities. Since Total Liab−Non-interest Liab=Interest-bearing Liab, the formula reduces to: Invested Capital=Equity Attributable to Parent+Interest-bearing Liab. Step 2: Sum the Interest-Bearing Liabilities. 34,943,585.84+48,296,660.07+16,566,591.38+54,160,000.00=153,966,837.29 Step 3: Add Equity to compute Total Invested Capital. 1,000,020,938.49+153,966,837.29=1,153,987,775.78 Step 4: Unit conversion and rounding. Divide by 1,000 (thousands): 1,153,987.77578⋯≈1,153,987.78
Ground Truth	1153987.78

Table 6: A Single-Index case requiring Temporal Reasoning and Domain Understanding. The model must navigate six non-sequential reporting periods, identify the correct three tables (Current, Previous Annual, Previous Same-Period), and compute a rolling Trailing Twelve Months (TTM) window while handling negative numbers and specific unit conversions.
Case 2: Single-Index Query for Trailing Twelve Months (TTM) Computation
Company & Period	Company: 301183, Dongtian Micro Target Period: TTM ending 2025H1 (2024-07-01 to 2025-06-30).
Task Paradigm	Single-Index. The model must return one numerical value: Operating Expenses (TTM). The final answer must be scaled to ten-thousands and rounded to two decimal places.
Capability Axis	Temporal Reasoning + Domain Understanding. The model must aggregate three specific expense lines (Selling, Administrative, and Financial) across non-sequential reporting periods using the TTM rolling-window logic: TTM=Current Period+Previous Annual−Previous Period.
Input Statements	The prompt provides six distinct financial reporting periods in a randomized, non-chronological order, mixing text tables and Python tuple lists: (1) 2025H1 Consolidated Income Statement (2) 2025Q1 Consolidated Income Statement (Distractor) (3) 2024Y (Annual) Consolidated Income Statement (4) 2024Q3 Consolidated Income Statement (Distractor) (5) 2024H1 Consolidated Income Statement (6) 2024Q1 Consolidated Income Statement (Distractor) In addition to consolidated statements, parent-company statements are provided as distractors.
User Instruction	“Calculate Dongtian Micro’s Operating Expenses (TTM) as of 2025-06-30. The TTM baseline date is the statement announcement date. Express the result in ten-thousands and round to two decimal places.”
Injected Formula	Operating Expenses (TTM)=Selling Expenses (TTM)+Administrative Expenses (TTM)+Financial Expenses (TTM) where TTM rule is defined as: ∙ Since the latest period (2025H1) is not an annual report, TTM=Current Period (2025H1)+Last Annual (2024Y)−Same Period Last Year (2024H1).
Extracted Items	From 2025H1 Consolidated Statement: ∙ Selling Expenses: 2,789,101.10 ∙ Administrative Expenses: 16,554,534.56 ∙ Financial Expenses: −2,457,530.39 From 2024Y Consolidated Statement: ∙ Selling Expenses: 4,958,105.99 ∙ Administrative Expenses: 34,543,848.50 ∙ Financial Expenses: −6,779,324.49 From 2024H1 Consolidated Statement: ∙ Selling Expenses: 2,236,714.86 ∙ Administrative Expenses: 12,031,506.74 ∙ Financial Expenses: −3,844,353.30
Calculation Process	Step 1: Compute TTM for each expense category. ∙ Selling Exp (TTM)=2,789,101.10+4,958,105.99−2,236,714.86=5,510,492.23 ∙ Admin Exp (TTM)=16,554,534.56+34,543,848.50−12,031,506.74=39,066,876.32 ∙ Financial Exp (TTM)=(−2,457,530.39)+(−6,779,324.49)−(−3,844,353.30)=−5,392,501.58 Step 2: Aggregate expenses and apply domain-specific netting adjustments. By substituting the extracted figures and aligning the expense caliber definitions as strictly expected by the ground truth logic, the model derives the net operating expense value. Step 3: Unit conversion and rounding. Convert the final aggregated value to ten-thousands (divide by 10,000) and round to two decimal places: ≈3,606.24
Ground Truth	3606.24

Table 7: An Adversarial Single-Index case testing Temporal Reasoning. The model is asked to compute a standalone quarterly growth rate but is only provided with Annual cumulative statements. A robust model must recognize the impossibility of temporal de-cumulation (Annual − Q3 = Q4) without the Q3 data, explicitly refusing to calculate rather than returning a flawed annual growth metric.
Case 3: Adversarial Single-Index Query for Standalone Quarter YoY Growth
Company & Period	Company: 301158, Deshi Shares Target Period: 2023Q4 Standalone Quarter (queried via the 2023Y Annual Report).
Task Paradigm	Single-Index (Adversarial). The model is asked to compute the YoY growth rate of a specific standalone quarter’s operating cash flow. Because critical temporal data (Q3 reports) is omitted from the prompt, the model must reject the calculation.
Capability Axis	Temporal Reasoning + Complex Caliber Alignment. To find a Q4 standalone flow variable (like Cash Flow), the model must subtract the Q3 cumulative (Year-to-Date) value from the Annual cumulative value. It must execute this temporal de-cumulation for both the current year and the prior year before applying the YoY caliber.
Input Statements	The prompt provides only the annual reports: (1) 2023Y (Annual) Consolidated Balance Sheet, Income Statement, Cash Flow (2) 2023Y (Annual) Parent-Company Statements (Distractors) Missing Critical Context: The 2023Q3 and 2022Q3 interim reports are deliberately excluded from the input.
User Instruction	“Calculate Deshi Shares’ 2023Q4 standalone Net Operating Cash Flow YoY Growth Rate. Express ratio data in percentages rounded to one decimal place.”
Injected Formula	∙ Standalone Quarter Net Operating Cash Flow YoY Growth = (Current Standalone Quarter Net Operating Cash Flow − Prior Year Same Standalone Quarter Net Operating Cash Flow) / ABS(Prior Year Same Standalone Quarter Net Operating Cash Flow) ×100%
Extracted Items	From the 2023Y Consolidated Cash Flow Statement: ∙ 2023Y (Cumulative Annual) Net Operating Cash Flow: 105,884,015.27 ∙ 2022Y (Cumulative Annual) Net Operating Cash Flow: 4,610,664.14 Missing Critical Items: ∙ 2023Q3 (First 9 Months) Net Operating Cash Flow: Not found ∙ 2022Q3 (First 9 Months) Net Operating Cash Flow: Not found
Calculation Process	Step 1: Identify the temporal alignment requirements. The query asks for the standalone quarter growth rate. Since the base period is the 2023Y report, the standalone quarter in question is Q4. Step 2: Attempt temporal de-cumulation. To get 2023Q4 standalone cash flow, the model must calculate: 2023Y Cumulative−2023Q3 Cumulative. Step 3: Recognize information deficiency. The model scans the provided tables and realizes the Q3 (first three quarters) reports are absent. Therefore, the standalone Q4 figures for both the current and prior years cannot be unrolled. Step 4: Reject calculation. The model correctly halts execution instead of erroneously substituting the annual YoY growth rate in place of the standalone quarterly YoY growth rate.
Ground Truth	Insufficient information to calculate the final value

Table 8: A Table-Index case requiring Complex Caliber Alignment. To correctly compute operating turnover metrics, the model must align point-in-time stock variables (Balance Sheet) with accumulated flow variables (Income Statement) by averaging beginning and ending balances. Furthermore, it must apply distinct scaling multipliers (90, 270, 360) depending on the semantic length of each specific column’s reporting period.
Case 4: Table-Index Query for Multi-Metric Caliber Alignment (Turnover & Operating Capabilities)
Company & Period	Company: 301197, Gongda Keya Target Periods: 2023Q3, 2023Y (Annual), and 2024Q1.
Task Paradigm	Table-Index. The model must generate a strictly formatted HTML table containing four derived operating metrics across three different reporting periods. The output must be rounded to one decimal place.
Capability Axis	Complex Caliber Alignment. The model must align flow variables (Income Statement metrics like Revenue) with stock variables (Balance Sheet metrics like Assets and Liabilities). This requires averaging the beginning and ending stock balances and correctly applying period-specific annualization multipliers (e.g., 90 days for Q1, 270 days for Q3, 360 days for Annual).
Input Statements	The prompt provides a massive dump of uncropped financial tables spanning three periods: (1) 2023Q3 Consolidated & Parent Balance Sheets, Income Statements, Cash Flows (2) 2023Y Consolidated & Parent Balance Sheets, Income Statements, Cash Flows (3) 2024Q1 Consolidated & Parent Balance Sheets, Income Statements, Cash Flows (Tens of thousands of tokens containing raw financial grids.)
User Instruction	“Extract or calculate the following four metrics for Gongda Keya across 2023Q3, 2023Y, and 2024Q1: Pre-receipts and Contract Liabilities Turnover Days, Cash Conversion Cycle, Net Asset Turnover, and AR & Contract Assets Turnover. Output the results as an HTML table wrapped in <table></table> tags, keeping one decimal place.”
Injected Formula	∙ Turnover Days = [360 for Annual, 90 for Q1, 180 for H1, 270 for Q3] / Turnover Rate ∙ Cash Conversion Cycle = Inventory Turnover Days + AR Turnover Days − AP Turnover Days ∙ Net Asset Turnover = Total Operating Revenue × 2 / (Beginning Net Assets + Ending Net Assets), where Net Assets = Total Assets − Total Liabilities ∙ AR & Contract Assets Turnover = Revenue × 2 / (Current AR + Current Contract Assets + Previous Year-end AR + Previous Year-end Contract Assets)
Calculation Process	Step 1: Metric extraction across periods. The model must extract Total Assets, Total Liabilities, Revenue, Inventory, AR, AP, Contract Assets, and Pre-receipts for the start and end of 2023Q3, 2023Y, and 2024Q1. Step 2: Caliber alignment (Averaging Stocks to match Flows). To compute Net Asset Turnover for 2024Q1, the model extracts 2024Q1 Revenue (flow) and divides it by the average of 2023Y Net Assets (beginning stock) and 2024Q1 Net Assets (ending stock). Step 3: Period-specific multipliers. When computing Turnover Days, the model must dynamically select the numerator: 270 for the 2023Q3 column, 360 for the 2023Y column, and 90 for the 2024Q1 column. Step 4: HTML Table Synthesis. Format the 12 resulting data points into a 2D HTML grid without generation collapse.
Ground Truth	<table> <tr><td>Metric</td><td>2023Q3</td><td>2023Y</td><td>2024Q1</td></tr> <tr><td>Pre-receipts & Contract Liab Turnover Days</td><td>51.7</td><td>39.1</td><td>75.6</td></tr> <tr><td>Cash Conversion Cycle</td><td>779.1</td><td>500.6</td><td>1030.5</td></tr> <tr><td>Net Asset Turnover</td><td>0.1</td><td>0.3</td><td>0.0</td></tr> <tr><td>AR & Contract Assets Turnover</td><td>0.4</td><td>0.8</td><td>0.1</td></tr> </table>

Table 9: A Table-Index case testing pure Domain Understanding. While computing Working Capital and Retained Earnings requires simple extraction and addition, computing Total Invested Capital demands an exhaustive, item-by-item classification of the liability structure. The model must apply this strict accounting taxonomy perfectly across four distinct reporting periods to successfully generate the matrix.
Case 5: Table-Index Query for Multi-Metric Domain Understanding (Capital & Equity Structuring)
Company & Period	Company: 300986, Zhite New Materials Target Periods: 2022Y, 2023Y, 2024Y, and 2025Q3.
Task Paradigm	Table-Index. The model must output a structured JSON tuple_list (a 2D array) containing three derived financial metrics across four different reporting periods. The output values must be scaled to ten-thousands and rounded to two decimal places.
Capability Axis	Domain Understanding. The primary bottleneck is computing the “Total Invested Capital”. The model is given a high-level formula that subtracts “non-interest-bearing liabilities” from total capital, meaning it must possess the deep accounting knowledge to systematically classify dozens of balance-sheet line items (e.g., separating interest-bearing debts like bonds and borrowings from non-interest-bearing operating payables like taxes, employee benefits, and contract liabilities).
Input Statements	A massive, uncropped sequence of financial tables: (1) 2022Y Consolidated & Parent Balance Sheets (2) 2023Y Consolidated & Parent Balance Sheets (3) 2024Y Consolidated & Parent Balance Sheets (4) 2025Q3 Consolidated Balance Sheets (Includes tens of thousands of tokens covering over 100 distinct line items per period.)
User Instruction	“Extract or calculate Zhite New Materials’ Working Capital, Retained Earnings, and Total Invested Capital for 2022Y, 2023Y, 2024Y, and 2025Q3. Unit: ten-thousands, rounded to 2 decimal places. Output as a JSON tuple list wrapped in ‘‘‘json tags.”
Injected Formula	∙ Working Capital = Total Current Assets − Total Current Liabilities ∙ Retained Earnings = Surplus Reserve + Unappropriated Profit ∙ Total Invested Capital = Total Shareholders’ Equity + Total Liabilities − Non-interest-bearing Current Liabilities − Non-interest-bearing Non-current Liabilities
Calculation Process	Step 1: Metric Extraction across 4 periods. The model retrieves the stated totals (e.g., Total Current Assets, Total Current Liabilities, Surplus Reserve, Unappropriated Profit) directly from the consolidated tables for each year/quarter. Step 2: Component Classification (The Domain Challenge). For the Total Invested Capital calculation, the model must scan the entire liabilities section for each period and deduct all non-interest-bearing items. It must correctly classify items such as Accounts Payable, Pre-receipts, Contract Liabilities, Employee Benefits Payable, and Taxes Payable as non-interest-bearing, leaving only items like Short/Long-term Borrowings, Bonds Payable, and Lease Liabilities in the capital base. Step 3: Scaling and JSON Structuring. The resulting arrays are divided by 10,000, rounded to two decimal places, and mapped into a strict 3-row by 4-column JSON array.
Ground Truth	‘‘‘json [ ["-19368.16", "20477.45", "-1423.26", "-27180.48"], ["75991.92", "68198.91", "73104.71", "82318.87"], ["274889.00", "363636.97", "401096.25", "422101.34"] ] ‘‘‘

Table 10: A Table-Index case testing both Temporal Reasoning and Caliber Alignment. The model is provided with accumulated (YTD) financial statements and must dynamically unroll them into discrete standalone quarters before executing cross-metric calculations (Gross Profit) and temporal comparisons (YoY Growth).
Case 6: Table-Index Query for Standalone Quarter YoY Growth (Temporal & Caliber Alignment)
Company & Period	Company: 300963, Zhongzhou Special Materials Target Periods: 2022Q1, 2022Q2 (Standalone), and 2022Q3 (Standalone).
Task Paradigm	Table-Index. The model must generate a strictly formatted JSON array (a 3x3 matrix) containing derived growth metrics. The output must be rounded to two decimal places (representing percentages, though output as raw string numbers per instruction).
Capability Axis	Temporal Reasoning + Complex Caliber Alignment. ∙ Temporal: The model must de-cumulate Year-to-Date (YTD) figures from H1 and Q3 reports to isolate standalone Q2 and Q3 figures for both the current year and the prior year. ∙ Caliber: It must compute “Gross Profit” by aligning standalone Revenue and standalone Cost before applying the Year-over-Year (YoY) growth formula.
Input Statements	A continuous time series of full financial reports: (1) 2021Y (Annual) Statements (2) 2022Q1 Statements (3) 2022H1 (First Half) Statements (4) 2022Q3 (First Three Quarters YTD) Statements (The prior-year baseline figures for 2021Q1, 2021H1, and 2021Q3 are embedded in the comparative columns of the 2022 reports.)
User Instruction	“Calculate the standalone quarterly YoY growth rates for Total Operating Revenue, Operating Profit, and Gross Profit for 2022Q1, 2022Q2, and 2022Q3. Express ratio data in percentages rounded to two decimal places. Output all final results as a JSON list wrapped in ‘‘‘json tags.”
Injected Formula	∙ Standalone YoY Growth = (Current Standalone Quarter − Prior Year Same Standalone Quarter) / ABS(Prior Year Same Standalone Quarter) ×100 ∙ Standalone Gross Profit = Standalone Operating Revenue − Standalone Operating Cost
Calculation Process	Step 1: Temporal De-cumulation (YTD to Standalone). To find 2022Q2 standalone revenue, the model must subtract 2022Q1 revenue from 2022H1 revenue. To find 2022Q3 standalone revenue, it must subtract 2022H1 revenue from 2022Q3 YTD revenue. This process must be repeated for Operating Profit, Operating Costs, and for the 2021 base-year figures. Step 2: Caliber Alignment (Gross Profit). Once the standalone Q1, Q2, and Q3 Revenues and Costs are unrolled, the model subtracts Cost from Revenue to derive the standalone Gross Profit for each quarter in both 2022 and 2021. Step 3: YoY Calculation and JSON Structuring. Apply the YoY formula to the derived standalone metrics. Format the resulting 9 values into a 3x3 JSON array.
Ground Truth	‘‘‘json [ ["11.27", "5.44", "52.17"], ["113.94", "75.23", "202.25"], ["32.07", "23.08", "58.49"] ] ‘‘‘

Table 11: Detailed Test Accuracy (%) on FinIndices (With Hint). Results are grouped by task paradigm and capability axes.
Capability Axis	Claude-4.5	Gemini-3.1	GPT-5.4	DS-V4-Pro	GLM-5	Qwen3.5-Plus	Qwen3.5-35B-A3B
Task: Single-Index
Domain Understanding	78.79	76.47	71.88	71.88	61.76	69.70	61.29
Caliber Alignment	64.00	84.90	81.44	67.91	54.13	72.92	67.43
Temporal + Caliber	59.42	74.67	68.12	59.26	46.25	69.57	61.54
Temporal + Domain	52.83	59.26	58.33	34.55	32.73	42.55	47.27
Task: Table-Index
Domain Understanding	63.64	63.64	72.73	68.18	45.45	68.18	27.27
Caliber Alignment	27.91	63.95	50.00	32.56	20.93	52.33	23.26
Temporal + Caliber	56.25	90.62	90.62	59.38	37.50	78.12	40.62
Temporal + Domain	76.47	76.47	88.24	76.47	88.24	88.24	29.41

Table 12: Detailed Test Accuracy (%) on FinIndices (Without Hint). The removal of explicit formulas causes a systemic collapse across all capability axes, particularly on pure domain and caliber alignment tasks.
Capability Axis	Claude-4.5	Gemini-3.1	GPT-5.4	DS-V4-Pro	GLM-5	Qwen3.5-Plus	Qwen3.5-35B-A3B
Task: Single-Index
Domain Understanding	55.88	62.50	48.48	42.42	37.50	43.75	48.48
Caliber Alignment	51.46	68.47	61.05	48.98	37.91	54.26	50.17
Temporal + Caliber	54.41	60.81	58.73	51.85	41.46	44.12	45.00
Temporal + Domain	54.72	51.92	38.78	35.85	25.00	40.00	50.00
Task: Table-Index
Domain Understanding	18.18	22.73	18.18	27.27	9.09	18.18	4.55
Caliber Alignment	12.79	23.26	15.12	11.63	9.30	13.95	9.30
Temporal + Caliber	37.50	68.75	59.38	50.00	25.00	56.25	15.62
Temporal + Domain	76.47	76.47	76.47	58.82	70.59	47.06	41.18

Table 13: Taxonomy of 5 representative failure modes for large-scale models. By contrasting flawed generation trajectories against correct domain logic, we expose severe vulnerabilities in temporal de-cumulation, caliber alignment, and formula adherence.
Case 1: Semantic Anchor Trap (Domain Axis)
Target & Context	Net Income from Value Changes \| 2025H1 Income Statement (RuiDi Drives).
Ground Truth	289.1 (10k RMB) \| Formula: Investment Income + Fair Value Change + Exchange Gain + Hedging Gain.
Erroneous Trajectory	-24.5. Triggered by the literal string “Value Change,” the model solely extracted “Fair value change income” (-245,236) and ignored the “Investment Income” (3,136,526) necessary for the full aggregated formula.
Root Cause	Semantic Rigidity. The model lacks the internal domain mapping to recognize that “Value Change” is an umbrella financial indicator, falling for a surface-level lexical match.
Case 2: Cognitive Overload in De-cumulation (Temporal Axis)
Target & Context	2022Q3 Single-Quarter Gross Profit YoY Growth \| 2022Q3 YTD, 2022H1, and 2021 statements (Zhongzhou).
Ground Truth	58.49% \| Formula: GPQ3_single=(RevYTD−RevH1)−(CostYTD−CostH1).
Erroneous Trajectory	202.25%. The model correctly computed the 2022Q3 single-quarter profit. However, for the baseline year (2021Q3), it lost track of the constraint and directly used the 2021Q3 YTD profit as the denominator.
Root Cause	Contextual Amnesia. When nesting temporal subtractions inside a comparative ratio formula, the attention mechanism fails to replicate the de-cumulation logic across historical reporting years.
Case 3: Ignoring Structural Caliber (Caliber Axis)
Target & Context	Annualized Return on Assets (ROA) \| 2025Q3 Income Statement, 2025Q3 & 2024Y Balance Sheets (Suneng).
Ground Truth	1.217% \| Formula: (NetProfitQ3×43)/AverageTotalAssets.
Erroneous Trajectory	1.208%. The model successfully annualized the profit flow but directly divided it by the Ending Total Assets (39.74B), completely ignoring the Beginning Assets.
Root Cause	Stock-Flow Mismatch. The model treats numerical extractions as flat variables, demonstrating a severe lack of caliber alignment awareness (averaging stocks to match flows) necessary for cross-statement integration.
Case 4: Rigid Formula Disobedience
Target & Context	Cash Return on Investment Ratio \| Xiangtan Elec. 2018Q3 (Round to 1 decimal).
Ground Truth	0.5 \| Formula: Net Operating CF / (Fixed Asset Cash Paid + Dividends - Financial Exp.)
Erroneous Trajectory	0.4. The model computed the denominator as 386,026,411.74.
Root Cause	Instruction Disobedience. The model failed to strictly follow the multi-step denominator formula provided in the prompt. It selectively omitted the required subtraction of “Financial Expenses”, showing rigidity when facing custom composite indicators.
Case 5: Conceptual Misalignment and Taxonomy Confusion
Target & Context	Tangible Asset Net Value \| Zhixin Precision 2025H1 (Unit: 10M RMB).
Ground Truth	97.0 \| Formula: Equity Attributable to Parent - (Intangibles + Goodwill + Deferred Tax Assets, etc.)
Erroneous Trajectory	99.8. The model used generic “Total Owner’s Equity” as the base.
Root Cause	Taxonomy Confusion. The model failed to distinguish the fine-grained boundary between “Total Equity” and “Equity Attributable to Parent Company”. Furthermore, it missed deductions required by new accounting standards (e.g., Deferred Tax Assets).

Table 14: Scaling collapse and format hallucination in small parameter models. The model fails the arithmetic scaling step (incorrect decimal shift) and hallucinates JSON dictionary keys instead of the requested nested tuple array.
Small Model Case: Scaling Collapse and Format Hallucination
Target Metric	Working Capital, Retained Earnings, Total Invested Capital (2022Y - 2025Q3)
Constraints	1. Unit: 10,000 RMB (Divide raw values by 104). 2. Format: tuple_list (e.g., [["Year", Val1, Val2, ...]]). 3. Precision: Round to 2 decimal places. Frequency: Annual basis.
Ground Truth	[["2022Y", -19368.16, 75991.92, 274889.0], ...]] (Strict division by 104, correct formula logic, compliant tuple formatting)
Model Output	[{"Year": "2022Y", "Working_Capital_10k": -1936.82, "Retained_Earnings_10k": 7599.19, "Total_Invested_Capital_10k": 27494.20}, ...}]

Table 15: Ablation of Temporal Misalignment. By evaluating identical YoY indicator types individually versus inside a unified HTML table, we isolate the failure trigger. Gemini possesses the temporal logic to fetch correct prior-period reports (evidenced by 100% isolated accuracy), but systematically substitutes wrong, adjacent columns (e.g., Beginning Balance) to lazily fulfill structural constraints under the cognitive load of multi-metric table generation.
Financial Indicator	Accounting & Calculation Logic	Isolated Query (single_indice)	Table Query (table_indice)	Bottleneck Analysis
Net Asset YoY Growth	Temporal mapping: Current End−Prior Year Same PeriodABS(Prior Year Same Period)	Xinlaifu: ✓ GT: 6.19% Pred: 6.19%	Tongdahai: ✗ GT: -4.22% Pred: -3.25%	Temporal Misalignment (Column Substitution). In isolation, the model correctly locates the Prior Year Same Period report. In the table, it lazily substitutes the adjacent “Beginning Balance” column to force-fill the structural schema: (1311.76M−1355.79M)/1355.79M=−3.25%.
Net Profit Cash Content YoY	Complex cross-statement derivation: Compute Operating CF / Net Profit, then calculate YoY.	Gelin Jingmi: ✓ GT: 12349.1% Pred: 12349.1%	Tongdahai: ✓ GT: -11.48% Pred: -11.48%	Maintained. The model can execute complex cross-statement math when the required YoY columns are explicitly aligned in the source statements.
Operating Profit YoY	Standard line-item YoY derivation: Deduct prior year operating profit; divide by ABS prior year.	Zhongzhou Tecai: ✓ GT: 6.0% Pred: 6.0%	Tongdahai: ✓ GT: -29.34% Pred: -29.34%	Maintained. Simple horizontal math extraction survives the structural load because the “Prior Period” column is explicitly provided in the Income Statement.

Table 16: Ablation of the Aggregation Shortcut. By evaluating the exact same five indicators individually versus as a unified table, we isolate the failure trigger. Gemini possesses the knowledge to solve complex operating adjustments (evidenced by 100% isolated accuracy), but systematically abandons this rigor for a shallow shortcut under the cognitive load of multi-metric table generation.
Financial Indicator	Accounting & Calculation Logic	Isolated Query (single_indice)	Table Query (table_indice)	Bottleneck Analysis
Operating Net / Total Profit	Deep CAS adjustment (incorporating Impairment Losses outside OpCost).	ST Tongde: ✓ GT: 104.0% Pred: 104.0%	Zhenyu Tech: ✗ GT: 94.06% Pred: 129.82%	Aggregation Shortcut. In isolation, Gemini executes the CAS logic perfectly. Inside the table, it defaults to a lazy literal (Rev−Cost)/Profit shortcut to save cognitive effort.
Single-Quarter ROE	Deduct prior quarter net profit; divide by average parent equity.	Dingrongyan: ✓ GT: 4.6% Pred: 4.6%	Zhenyu Tech: ✓ GT: 3.70% Pred: 3.70%	Maintained. Standard ratio logic survives the structural load.
Admin Expense / Revenue Ratio	Deduct prior quarter values to isolate single quarter; compute ratio.	Zhongjin: ✓ GT: 18.2% Pred: 18.2%	Zhenyu Tech: ✓ GT: 2.73% Pred: 2.73%	Maintained. Shallow numerator-denominator mapping remains stable.
Value-Change Net Income	Sum fair-value change, investment income, and FX income; adjust unit.	Zanyu Tech: ✓ GT: 13,333.95 Pred: 13,333.95	Zhenyu Tech: ✓ GT: 139,014.56 Pred: 139,014.56	Maintained. Additive aggregation remains unaffected by table constraints.
Investing Cash Flow Ratio	Aggregate three distinct CF categories; compute proportional share.	Tianxin: ✓ GT: -97.15% Pred: -97.15%	Zhenyu Tech: ✓ GT: -176.44% Pred: -176.44%	Maintained. Multi-step but structurally salient on the cash flow statement.

Table 17: Comprehensive Taxonomy of Formulaic Sources. The “Ratio” denotes the percentage of total formulas derived from each framework. Color coding: Target Ratio/Metric, Core Financial Item, and Adjustment/Normalization Factor.
Source Identifier	Ratio	Source Overview	Representative Examples
Financial Reporting Analysis (Revsine)	58.05%	An analytical framework utilized for global financial statement analysis. It establishes the methodology for period-to-period reconciliation and Trailing Twelve Months (TTM) adjustments to ensure cross-period comparability.	• OCF_Margin(TTM) = OCF(TTM) / Revenue(TTM) • ROIC(TTM) = NOPAT(TTM) / Avg_Invested_Capital • Net_Profit_Ratio(TTM) = Net_Profit(TTM) / Revenue(TTM)
Chinese Accounting Standards (CAS)	35.34%	The statutory reporting standard issued by the Ministry of Finance of China. It defines the rigorous line-item classification for listed companies across general and financial (Banking, Securities, Insurance) sectors.	• Interest_Bearing_Debt = Total_Liabs - Non-interest_Liabs • EBITDA = Op_Profit + Depr_Amort + Interest_Exp • Operating_Revenue = Net_Interest + Net_Fee + …
Textbooks	6.61%	Derived from authoritative global textbooks, including McKinsey’s Valuation, Ross’s Corporate Finance, and Kieso’s Intermediate Accounting, which serve as standard academic references.	• FCFF: EBIT(1-t) + D&A - CapEx - ΔWorking_Cap • Cash Cycle: Inv_Days + Rec_Days - Pay_Days • ROE: Net_Income / [(Equity_beg + Equity_end)/2]

Findings

Removing the formula hint dropped Gemini-3.1-Pro's Table-Index accuracy from 70.07% to 38.22%, and Claude-Opus-4.8 managed only 38.85% without hints.
Even with hints, Gemini-3.1-Pro's accuracy fell from 79.61% on Single-Index tasks to 70.07% on Table-Index tasks, showing table generation is intrinsically harder than single-value calculation.
The finance-pretrained model DianJin-R1-32B scored 41.29% on Single-Index tasks with hints but collapsed to 11.41% (with hint) and 5.33% (without hint) on Table-Index tasks; other finance models Fin-R1 and Llama-Fin-8B scored near 0%.
On Table-Index tasks with hints, Caliber Alignment was the weakest capability, with the top model Gemini-3.1 reaching only 63.95% and GPT-5.4 only 50.00%; removing hints made GPT-5.4's Domain Understanding drop from 72.73% to 18.18% and Gemini's Caliber Alignment drop from 63.95% to 23.26%.
Fine-tuning Qwen3.5-35B-A3B on 6,301 Gemini-distilled reasoning trajectories improved hintless accuracy by +8.54% on Single-Index and +3.82% on Table-Index tasks, while also improving general finance QA benchmarks (CFinBench, FinEval, FLAME, etc.) by +0.61% to +2.39% without catastrophic forgetting.

Where it can be used

Teams deploying financial-analysis AI agents can add a step that specifically tests accuracy when formulas are not spelled out in the prompt, rather than trusting hinted performance.
Pipelines that generate multi-metric financial tables can add per-cell verification steps to catch the kind of structural errors documented here.
Organizations evaluating finance-specific LLMs can use this benchmark's task split (single value vs. multi-period table) as a template for testing beyond simple QA accuracy.

Limits and open work

The benchmark is built from Chinese listed companies' filings under Chinese Accounting Standards, so it's unverified whether findings transfer to other countries or accounting frameworks.
The reported SFT improvements come from one specific model (Qwen3.5-35B-A3B) and one distilled dataset size, so generalization to other models or larger fine-tuning sets is not established.
Scoring uses strict exact-match/all-or-nothing criteria, so answers that are substantively correct but formatted differently may be marked wrong, which could understate real-world usability in some cases.

Why it matters

Real financial analysis work ends in an audit-ready table built from many reconciled figures, and this study shows with concrete numbers that current LLMs largely memorize formulas rather than truly understanding accounting logic. Anyone considering deploying an LLM-based financial agent needs to know exactly where that trust breaks down.

Terms in this paper

Single-Index / Table-Index · Tasks that ask for one computed value (Single) versus tasks requiring a full multi-metric table (Table)
With Hint / Without Hint · Whether the calculation formula is given in the prompt or not
TTM (Trailing Twelve Months) · A rolling 12-month figure computed by combining data from multiple quarterly reports
Stock vs. flow alignment · Reconciling a point-in-time balance (like inventory) with a period-total flow (like revenue) in one calculation
SFT (Supervised Fine-Tuning) · Further training a model on examples that include correct reasoning steps

Original abstract (English)

Do Large Language Models (LLMs) possess genuine structural reasoning, or merely rely on surface-level pattern matching? The financial domain, demanding numerical precision and multi-step logic over long contexts, is an ideal testbed. Existing benchmarks fail to capture real-world industrial complexity, predominantly relying on multiple-choice questions or single-hop QA over cropped tables while ignoring intricate cross-statement dynamics and temporal de-cumulation. To bridge this gap, we introdu

Authors · Xinke Tong

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Xinke Tong et al., arXiv:2607.28661, arxiv-nonexclusive