Reliable Financial Named Entity Recognition under Domain Shift
An AI's confidence trained on formal filings turns unreliable once it reads tweets
Researchers took named-entity recognition AI trained on SEC filings and tested it on financial news and general social media, finding that not just accuracy but the AI's own confidence scores broke down under this shift. Whole-sentence probability, the best confidence signal in-domain, became untrustworthy out of domain, while entity-span probability and self-consistency (agreement across repeated generations) held up better. Still, on the extreme social-media shift, no confidence signal could reliably identify a safe subset of predictions to automate.
METAL MEDIA explanatory visual
An AI's confidence trained on formal filings turns unreliable once it reads tweets
- 01Trained a BERT tagger and small language models (Qwen2.5 0.5B/1.5B fine-tuned with LoRA) on SEC filings, then tested them on financial news and general-topic Twitter data across three seeds with bootstrap confidence intervals
- 02Compared five inference-time confidence signals: whole-output probability, token probability, entity-span probability, entity-type probability, and self-consistency across repeated samples
- 03Whole-output probability was the strongest error detector in-domain but degraded sharply under domain shift; entity-span probability and self-consistency were more robust, with self-consistency also better calibrated without any post-hoc fixing
- 04Letting the AI automatically answer only the top 40% most-confident sentences cut in-domain sentence error from 34.3% to under 2%, still helped on financial news, but found no usefully large safe subset on the extreme social-media data
- 05Proposes a staged deployment approach: detect severe domain shift first, then apply confidence-based automation only within domains the system can reliably handle
What they did
- Trained a BERT tagger and small language models (Qwen2.5 0.5B/1.5B fine-tuned with LoRA) on SEC filings, then tested them on financial news and general-topic Twitter data across three seeds with bootstrap confidence intervals
- Compared five inference-time confidence signals: whole-output probability, token probability, entity-span probability, entity-type probability, and self-consistency across repeated samples
- Whole-output probability was the strongest error detector in-domain but degraded sharply under domain shift; entity-span probability and self-consistency were more robust, with self-consistency also better calibrated without any post-hoc fixing
- Letting the AI automatically answer only the top 40% most-confident sentences cut in-domain sentence error from 34.3% to under 2%, still helped on financial news, but found no usefully large safe subset on the extreme social-media data
- Proposes a staged deployment approach: detect severe domain shift first, then apply confidence-based automation only within domains the system can reliably handle
| Split | Domain | Sent. | Ent. | PER | ORG | LOC |
|---|---|---|---|---|---|---|
| Train | FIN (filings) [27] | 1014 | 980 | 648 | 175 | 157 |
| Valid | FIN (filings) [27] | 150 | 177 | 97 | 68 | 12 |
| Test | FIN (filings) [27] | 299 | 295 | 201 | 56 | 38 |
| Test | FiNER-ORD (news) [28] | 300 | 322 | 78 | 151 | 93 |
| Test | TweetNER7 (tweets) [30] | 300 | 619 | 381 | 135 | 103 |
| Model | Domain | P | R | F1 | Halluc.% | ECE |
|---|---|---|---|---|---|---|
| BERT-base | FIN | 66.7±2.3 | 74.2±1.4 | 70.3±1.5 | – | 0.102±0.016 |
| BERT-base | FiNER-ORD | 39.1±0.3 | 37.6±2.3 | 38.3±1.1 | – | 0.071±0.017 |
| BERT-base | TweetNER7 | 35.2±5.2 | 18.4±1.7 | 24.1±2.5 | – | 0.074±0.007 |
| Qwen-0.5B | FIN | 40.5±0.4 | 34.7±2.1 | 37.4±1.3 | 3.6±2.4 | 0.362±0.009 |
| Qwen-0.5B | FiNER-ORD | 30.7±1.3 | 18.0±4.4 | 22.6±3.8 | 8.2±0.7 | 0.407±0.005 |
| Qwen-0.5B | TweetNER7 | 30.8±0.7 | 23.7±4.2 | 26.7±2.8 | 5.6±1.1 | 0.370±0.007 |
| Qwen-1.5B | FIN | 47.7 | 44.7 | 46.2 | 4.3 | 0.322 |
| Qwen-1.5B | FiNER-ORD | 56.3 | 48.4 | 52.1 | 3.6 | 0.227 |
| Qwen-1.5B | TweetNER7 | 44.9 | 50.9 | 47.7 | 6.8 | 0.358 |
| Signal | FIN | FiNER-ORD | TweetNER7 | |||
|---|---|---|---|---|---|---|
| AUROC | ECE | AUROC | ECE | AUROC | ECE | |
| Sequence prob. | 0.839±0.014 | 0.512±0.005 | 0.661±0.015 | 0.593±0.011 | 0.626±0.019 | 0.586±0.006 |
| Token prob. | 0.823±0.012 | 0.531±0.004 | 0.666±0.019 | 0.615±0.013 | 0.605±0.018 | 0.611±0.006 |
| Span prob. | 0.801±0.004 | 0.362±0.009 | 0.690±0.030 | 0.407±0.005 | 0.721±0.011 | 0.370±0.007 |
| Type prob. | 0.492±0.019 | 0.506±0.004 | 0.536±0.022 | 0.533±0.019 | 0.511±0.010 | 0.573±0.002 |
| Self-consistency | 0.690±0.019 | 0.116±0.029 | 0.671±0.048 | 0.120±0.015 | 0.682±0.017 | 0.100±0.008 |
| Signal | 0.5B (3 seeds) | 1.5B (1 seed) | ||||
|---|---|---|---|---|---|---|
| FIN | FiNER | Tweet | FIN | FiNER | Tweet | |
| Sequence prob. | 0.839 | 0.661 | 0.626 | 0.764 | 0.720 | 0.637 |
| Span prob. | 0.801 | 0.690 | 0.721 | 0.819 | 0.729 | 0.638 |
| Self-consistency | 0.690 | 0.671 | 0.682 | 0.735 | 0.744 | 0.663 |
| Domain | Signal | P@100% | P@80% | P@60% |
|---|---|---|---|---|
| FIN | Seq. prob. | 40.5±0.4 | 49.0±0.8 | 58.4±0.6 |
| FIN | Span prob. | 40.5±0.4 | 49.2±0.3 | 56.4±0.6 |
| FIN | Self-consistency | 40.5±0.4 | 47.7±0.4 | 52.8±1.0 |
| FiNER | Seq. prob. | 30.7±1.3 | 34.9±3.0 | 40.1±1.3 |
| FiNER | Span prob. | 30.7±1.3 | 35.3±1.7 | 40.8±2.6 |
| FiNER | Self-consist. | 30.7±1.3 | 34.9±3.0 | 41.4±2.3 |
| Tweet | Seq. prob. | 30.8±0.7 | 34.2±1.0 | 36.8±2.9 |
| Tweet | Span prob. | 30.8±0.7 | 36.6±1.1 | 42.7±1.1 |
| Tweet | Self-consist. | 30.8±0.7 | 35.5±1.7 | 40.5±1.5 |
| Filtered | Relaxed | |
|---|---|---|
| Sentences dropped (of 2807) | 2179 | 0 |
| Evaluated (cap 300) | 300 | 300 |
| Gold PER/ORG/LOC | 619 | 443 |
| Ignored entities | 0 | 488 |
| Encoder F1 | 24.1±2.5 | 19.5±1.0 |
| Encoder MSP AUROC | 0.615±0.054 | 0.650±0.047 |
| Qwen-0.5B F1 | 29.7 | 29.1 |
| Qwen-0.5B span AUROC | 0.725 | 0.715 |
Why it matters
In finance, where extraction errors feed into compliance and risk pipelines, accuracy scores alone can't tell you when it's safe to trust an AI's output automatically. This study shows that an AI's self-reported confidence can itself fail silently under domain shift, and offers concrete guidance on which confidence signals to trust and when.
Terms in this paper
- Named Entity Recognition (NER) · an AI task that finds and labels names of people, organizations, and locations in text
- distribution shift · when the style or topic of real-world input text differs from the data the AI was trained on
- selective prediction · letting an AI withhold an answer when unsure and send that case to a human reviewer instead
- self-consistency · a confidence measure based on how often an AI gives the same answer across multiple generated outputs for the same input
- LoRA · a lightweight fine-tuning method that adjusts only a small set of added parameters instead of retraining the whole model
- AUROC / ECE · AUROC measures how well a confidence score separates correct from incorrect predictions; ECE measures how well the stated confidence matches actual accuracy
Original abstract (English)
Financial AI systems often train information extractors on one textual register and deploy them across filings, news, and user-generated content, while standard F1 scores do not indicate which predictions remain safe to automate when the input distribution changes. We study confidence estimation and selective prediction for financial named entity recognition (NER) on a three-tier stress test spanning SEC filings, financial news, and general-topic social media as an extreme out-of-domain condition. We evaluate a BERT tagger and LoRA-tuned Qwen2.5-0.5B/1.5B models using five inference-time confidence signals, three training seeds, and bootstrap intervals. Confidence rankings themselves change under distribution shift: whole-output probability is the strongest in-domain error detector but deteriorates out of domain, whereas entity-span probability and self-consistency are more robust; self-consistency is also better calibrated without post-hoc fitting. Abstention reduces sentence error from 34.3% to below 2% on the highest-confidence 40% of in-domain inputs and remains useful on financial news, but recovers no usefully large clean subset under the extreme social-media shift. These results motivate a staged deployment strategy that detects severe distribution shift upstream before applying prediction-level confidence gating.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctly
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears
Latest from METAL MEDIA
Figures: Zihao Zheng et al., arXiv:2608.19558, arxiv-nonexclusive