Temporal Multi-Signal Fusion for Token-Level Hallucination Detection

What they did

1. Current token-level hallucination detectors score each word independently and fail exactly when the generating model is confidently wrong.
2. The authors extract a 33-dimensional feature vector per token combining text statistics, Natural Language Inference (NLI) entailment scores, and language-model surprisal, then feed the whole sequence to a Bidirectional Gated Recurrent Unit (BiGRU) that reads context in both directions.
3. Using only the generated text and external models (no access to the generating model's internals), BiGRU reached 0.840 AUC on RAGTruth versus 0.730 for an independent logistic-regression baseline (10 seeds, p=0.002) an 11-point gain.
4. A controlled breakdown attributed the gain to temporal order (44%), sequence-wide aggregation (24%), and nonlinear model capacity (32%).
5. Performance degraded by less than 4% when tested on text from language models unseen during training, and switching a CRF model's scoring method to forward-backward marginals recovered up to 17.9 AUC points.

Why it matters

Because hallucinations tend to spread token by token once they start, catching that pattern lets this detector work on closed-source models where internal signals aren't available. That matters for real deployments -- like retrieval-augmented generation in healthcare, legal, or financial settings -- where a single fabricated claim can propagate downstream.

Terms in this paper

• hallucination · fluent but factually wrong text generated by an AI language model
• NLI (Natural Language Inference) · a technique that judges whether one sentence entails, contradicts, or is neutral toward another
• surprisal · a measure of how unexpected a word is according to a language model's predictions
• BiGRU (Bidirectional Gated Recurrent Unit) · a neural network that reads a sequence both forward and backward to capture context
• AUC · a score measuring how well a model distinguishes correct from incorrect cases, higher is better
• CRF (Conditional Random Field) · a structured prediction method that models dependencies between neighboring labels

Figures we cannot republish

• Figure 2: Decomposition of BiGRU’s 11-point advantage over LogReg. Temporal order is the largest contributor (44%).
• Figure 3: Feature dynamics aligned to hallucination span onset (1,136 spans). LM entropy spikes at onset; context overlap drops; NLI contradiction rises gradually; BiGRU probability accumulates monotonically.
• Figure 4: AUC distribution across 10 seeds for key architectures. Temporal models (BiGRU, BiLSTM) consistently outperform non-temporal baselines (LogReg, MLP).
• Figure 5: Token AUC across all architecture–signal combinations (single seed). Non-CRF temporal models (BiGRU, BiLSTM) consistently outperform baselines across all signal configurations.
• Figure 6: Leave-one-out cross-model generalization (all signals, 10 seeds). GPT-4 is the hardest transfer target; LLaMA-2-70B is the easiest.
• Figure 7: Waterfall decomposition of signal contributions to BiGRU’s AUC. NLI and LM features provide complementary gains beyond text features alone.
• Figure 8: Precision-recall curves for all non-CRF architectures (all signals, seed 42). The low overall precision reflects RAGTruth’s 5.6% hallucination rate.
• Figure 9: Per-task AUC comparison (all signals). The temporal advantage of BiGRU over LogReg is largest for Data2txt.
• Figure 10: Token AUC by sequence length quartile. The BiGRU–LogReg gap grows from +8.1 (Q1) to +12.2 (Q4).
• Figure 11: CRF scoring methods compared. Forward-backward marginals recover up to +17.9 AUC points over softmax scoring.
• Figure 12: Natural counterfactual: same query, different LLMs, one hallucinating. NLI contradiction separates the two; LM entropy does not; BiGRU integrates both.
• Figure 13: Bidirectional dataset transfer. Reverse transfer (PsiloQA to RAGTruth) outperforms forward despite fewer training examples.
• Figure 14: Architecture comparison (cosine schedule). All temporal architectures with full-sequence scope converge to ∼0.84 AUC. With plateau scheduling (Table 21), the ceiling rises to 0.845.

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversightarXiv:2608.192162026-08-21
• PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappearsarXiv:2608.197462026-08-21
• Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based onearXiv:2608.195262026-08-21

Latest from METAL MEDIA

• NCT 127 drop BLINGY, their first full album in two years