FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systems
AI often names the right cause of a financial mismatch without ever finding the proof for it
A new benchmark called FinRCA-Bench tests AI systems that diagnose why company transactions fail to reconcile across invoices, payments, and bank records. Swapping only the retrieval method under the same language model pushed accuracy from 2.05% to 72.44%, showing that finding the right records mattered far more than the model's reasoning. Even correct answers were frequently unsupported: strict evidence-contract accuracy was just 5.72%.
METAL MEDIA explanatory visual
AI often names the right cause of a financial mismatch without ever finding the proof for it
- 01The benchmark uses 2,250 synthetic accounts-payable-to-bank reconciliation cases across 14 operational tables, including 1,500 cases with injected failures across 15 cause categories and 750 legitimate or tricky 'hard negative' cases.
- 02Instead of retrieving records by text similarity (Dense RAG), the study introduces Typed Provenance Graph Retrieval (TPGR), which follows only real, persisted transaction relationships between records, and compares both under the identical downstream reasoning model.
- 03Changing only the retrieval method raised macro required-record recall from 0.83% to 77.70% and exact 16-class accuracy from 2.05% to 72.44%, while structured baselines (rule-based system: 84.97%, classical machine learning: 95.44%) already outperformed the LLM-based system.
- 04Of 437 cases where the model got the right answer, 311 (71.17%) did so without meeting the required evidence contract, and only 5.72% passed the strictest evidence check; 95 of the remaining errors were traced to retrieval failure versus only 15 to genuine reasoning failure.
What they did
- The benchmark uses 2,250 synthetic accounts-payable-to-bank reconciliation cases across 14 operational tables, including 1,500 cases with injected failures across 15 cause categories and 750 legitimate or tricky 'hard negative' cases.
- Instead of retrieving records by text similarity (Dense RAG), the study introduces Typed Provenance Graph Retrieval (TPGR), which follows only real, persisted transaction relationships between records, and compares both under the identical downstream reasoning model.
- Changing only the retrieval method raised macro required-record recall from 0.83% to 77.70% and exact 16-class accuracy from 2.05% to 72.44%, while structured baselines (rule-based system: 84.97%, classical machine learning: 95.44%) already outperformed the LLM-based system.
- Of 437 cases where the model got the right answer, 311 (71.17%) did so without meeting the required evidence contract, and only 5.72% passed the strictest evidence check; 95 of the remaining errors were traced to retrieval failure versus only 15 to genuine reasoning failure.
Why it matters
For anyone deploying AI in financial operations, a correct answer alone doesn't mean the system actually found the records that justify it, which matters when auditors or finance teams need to trust and trace a diagnosis. This work gives concrete numbers showing that retrieval quality, not model intelligence, is often the real bottleneck.
Terms in this paper
- RAG (retrieval-augmented generation) · a method where an AI first fetches relevant records before generating an answer
- TPGR (Typed Provenance Graph Retrieval) · a retrieval method that follows only predefined, real transaction links between records instead of using similarity scoring
- hard negative · a case that looks suspicious but is actually legitimate, included to test whether the system overreacts
- macro recall · the average, across all cases, of how completely the needed records were retrieved
- provenance · the traceable chain showing where a record came from and how it genuinely connects to others
Original abstract (English)
Large language models are increasingly used to support financial operations, but their apparent reasoning performance can depend on whether they receive the right evidence. In financial reconciliation, the evidence needed for diagnosis is distributed across invoices, purchase orders, approvals, allocations, payments, ledger entries, and bank activity, linked by transactional relationships rather than textual similarity. End-to-end accuracy can therefore conflate evidence access with reasoning quality. We introduce FinRCA-Bench, a deterministic synthetic benchmark of 2,250 accounts-payable-to-bank reconciliation cases spanning 14 operational tables, including 1,500 injected failures across 15 causal categories and 750 legitimate or hard-negative cases. Root-cause labels and record-level evidence contracts are hidden from the model, allowing retrieval to be evaluated independently of answer correctness. We compare Rules/SQL, classical machine learning, dense semantic retrieval, deterministic relational expansion, and Typed Provenance Graph Retrieval (TPGR), a typed traversal restricted to persisted transaction relationships. Rules/SQL reaches 84.97% held-out exact accuracy and classical ML reaches 95.44%. Holding the reasoning model, prompt, and generation settings fixed while changing only retrieval increases macro required-record recall from 0.83% to 77.70% and exact 16-class accuracy from 2.05% to 72.44%. Structural retrieval failures outnumber reasoning failures with sufficient retrieval by 95 to 15; 254 correct predictions occur despite incomplete retrieval, and strict returned-evidence contract accuracy is only 5.72%. On FinRCA-Bench, retrieval architecture strongly shapes observed AI-system performance, and a correct root-cause label is a weak proxy for an auditable diagnosis.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears
- Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based one