StocksTalk: A Voice-Enabled Conversational Agent for Structured Query Generation over Web Data
A voice assistant that turns spoken stock-screening requests into checked SQL queries
StocksTalk is a conversational system that converts spoken investment screening requests, like 'large-cap IT stocks with P/E below 25,' into executable SQL queries over real financial data. It chains streaming speech recognition, retrieval-augmented constraint extraction, schema-grounded LLM SQL generation, rule-based validation, and a human confirmation step inside one dashboard. On a 150-prompt benchmark, the full pipeline beat a plain GPT-4o baseline by wide margins on logical consistency and multi-turn stability.
METAL MEDIA explanatory visual
A voice assistant that turns spoken stock-screening requests into checked SQL queries
- 01Built a pipeline that turns spoken investment constraints (P/E ratio, dividend yield, debt-to-equity, etc.) into SQL queries executed against live financial data from Screener.in
- 02Designed a four-stage architecture—streaming speech-to-text (ElevenLabs), retrieval-augmented constraint extraction, schema-grounded GPT-4o SQL generation, rule-based validation, and human-in-the-loop verification—exposing every intermediate result on a dashboard so users can confirm or correct each step before execution
- 03Curated and plans to publicly release FinScreenBench, a benchmark of 150 spoken financial screening prompts (300 audio recordings total) across three strategy categories—growth, dividend, value—and two recording conditions, clean and noisy
- 04On metrics covering constraint extraction accuracy, SQL executability, logical consistency, query edit distance, and multi-turn stability, the full system with human verification outperformed a plain GPT-4o baseline by 27.8 percentage points on logical consistency and 37.4 points on multi-turn stability
- 05An ablation study showed each component fixes a different failure mode: retrieval-augmented generation mainly helps constraint extraction, constrained decoding mainly helps executability, and rule-based validation mainly helps logical consistency; removing human verification let errors persist across turns in 34% of multi-turn cases, versus just 9% with it
What they did
- Built a pipeline that turns spoken investment constraints (P/E ratio, dividend yield, debt-to-equity, etc.) into SQL queries executed against live financial data from Screener.in
- Designed a four-stage architecture—streaming speech-to-text (ElevenLabs), retrieval-augmented constraint extraction, schema-grounded GPT-4o SQL generation, rule-based validation, and human-in-the-loop verification—exposing every intermediate result on a dashboard so users can confirm or correct each step before execution
- Curated and plans to publicly release FinScreenBench, a benchmark of 150 spoken financial screening prompts (300 audio recordings total) across three strategy categories—growth, dividend, value—and two recording conditions, clean and noisy
- On metrics covering constraint extraction accuracy, SQL executability, logical consistency, query edit distance, and multi-turn stability, the full system with human verification outperformed a plain GPT-4o baseline by 27.8 percentage points on logical consistency and 37.4 points on multi-turn stability
- An ablation study showed each component fixes a different failure mode: retrieval-augmented generation mainly helps constraint extraction, constrained decoding mainly helps executability, and rule-based validation mainly helps logical consistency; removing human verification let errors persist across turns in 34% of multi-turn cases, versus just 9% with it
| System | CEA (%) | EX (%) | LCR (%) | QED | MTS (%) | LAT (s) |
|---|---|---|---|---|---|---|
| GPT-4o (plain) | 63.4 | 81.2 | 54.7 | 8.3 | 51.2 | 1.5±0.1 |
| GPT-4o + RAG | 79.8 | 88.6 | 67.3 | 5.1 | 68.4 | 2.1±0.2 |
| GPT-4o + RAG + Validation | 88.3 | 96.9 | 91.2 | 2.8 | 82.7 | 2.3±0.2 |
| StocksTalk (full) | 91.2 | 97.5 | 93.8 | 2.1 | 88.6 | 3.1±0.4 |
| Metric | Clean Input | Noisy Input |
|---|---|---|
| Constraint Extraction Acc. (%) | 91.2 | 78.4 |
| SQL Executability (%) | 97.5 | 89.3 |
| Logical Consistency Rate (%) | 93.8 | 82.1 |
| Query Edit Distance (tokens) | 2.1 | 5.7 |
| Multi-turn Stability (%) | 88.6 | 74.3 |
| Avg. End-to-End Latency (s) | 3.1±0.4 | 3.6±0.7 |
| Configuration | CEA (%) | EX (%) | LCR (%) |
|---|---|---|---|
| Full system | 91.2 | 97.5 | 93.8 |
| w/o RAG retrieval | 74.3 (−16.9) | 95.1 (−2.4) | 87.2 (−6.6) |
| w/o validation layer | 90.8 (−0.4) | 96.9 (−0.6) | 71.4 (−22.4) |
| w/o constrained decoding | 88.5 (−2.7) | 79.2 (−18.3) | 84.3 (−9.5) |
| w/o human-in-the-loop | 89.1 (−2.1) | 94.3 (−3.2) | 80.6 (−13.2) |
| Prompt Type | w/o HITL | w/ HITL | Gain |
|---|---|---|---|
| Growth-focused | 94.2 | 98.6 | +4.4 |
| Dividend-oriented | 91.7 | 97.3 | +5.6 |
| Value-based | 88.3 | 96.8 | +8.5 |
| Overall | 91.4 | 97.5 | +6.1 |
Why it matters
Turning complex spoken conditions directly into runnable database queries is directly applicable to banks, brokerages, and any service building natural-language search interfaces. The design choice of exposing intermediate steps for user correction offers a concrete template for where humans should step in when an AI system might be wrong.
Terms in this paper
- RAG (retrieval-augmented generation) · an AI technique that looks up relevant documents or knowledge before generating an answer
- SQL · the query language used to search and filter data stored in a database
- human-in-the-loop · a process where a person reviews or corrects an AI's output before it is acted on
- ASR (automatic speech recognition) · technology that converts spoken audio into text
- schema-grounded prompting · giving an AI the structure of a database table so it only generates queries valid for that structure
Original abstract (English)
StocksTalk is a voice-enabled conversational system for transforming spoken financial screening requests into executable and validated structured queries over real-world market data. The system combines streaming speech recognition, retrieval-augmented constraint extraction, schema-grounded LLM-based SQL generation, rule-based validation, and human-in-the-loop verification within an interactive dashboard. Unlike traditional template-driven financial assistants, StocksTalk exposes intermediate reasoning artifacts, including extracted constraints, normalized financial metrics, operator grounding, and generated queries, allowing users to inspect and refine each stage before execution. To evaluate the system, we curate a benchmark of 150 spoken financial prompts spanning multiple investment strategies and input noise conditions. Experimental results show that retrieval grounding, constrained query generation, and interactive verification substantially improve constraint extraction accuracy, SQL executability, logical consistency, and multi-turn stability compared to baseline LLM-based approaches. StocksTalk demonstrates how transparent, voice-driven interfaces can bridge natural language interaction and structured financial analysis, providing an effective framework for conversational stock screening and decision support.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctly
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears
Latest from METAL MEDIA
Figures: Akshat Parmar et al., arXiv:2608.18105, CC BY 4.0