Outcome Monitors: Recovery Affordances for Silent Tool Failures

arXiv:2608.193032026-08-21

A no-force warning note that just says 'here's what else you can try' cuts AI agents' silent-failure blind spot in half

AI agents notice obvious tool failures like timeouts, but a cached error page or a negative price that arrives in the expected format can be swallowed as fact. The researchers built Outcome Monitors, which check tool results against rules mined from clean past runs or public API schemas, and when a rule is broken, attach a nonbinding receipt naming the violated property and listing usable recovery tools, without blocking or fixing anything themselves. Across multiple models and benchmarks, task completion rose substantially, and the effect traced specifically to the recovery-tool list rather than to extra diagnostic detail.

METAL MEDIA explanatory visual

A no-force warning note that just says 'here's what else you can try' cuts AI agents' silent-failure blind spot in half

01Problem: agents can spot a tool timeout, but a result that looks properly formatted yet is actually wrong (a cached error page, a negative price) tends to get accepted as true, a class of 'silent tool failures'
02Method: rules called outcome contracts are mined automatically from task-disjoint clean execution traces or derived from public API schemas; when a tool result violates one, the original result is kept unchanged and a nonbinding receipt is appended naming the violated property and listing publicly available recovery tools, leaving the choice of what to do entirely to the agent
03Result: on ToolMaze, completion rose from 10.9% to 28.1% averaged across four models from two provider families, and the gain replicated in a third provider's model (MiniMax M3); on tau-bench retail it improved by 14.0 and 12.0 percentage points across two difficulty tiers
04Verification: removing just the recovery-tool list from the receipt eliminated the gain, and restoring it brought the gain back, while adding more diagnostic detail or changing the timing of the warning made no detectable difference, showing the actionable tool list is what actually drives the effect
05Limitation: on a suite built from a real published catalog of production incidents, detection accuracy dropped to 46% for failure types outside the mined vocabulary, though the system still delivered results and overall completion was unaffected

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Problem: agents can spot a tool timeout, but a result that looks properly formatted yet is actually wrong (a cached error page, a negative price) tends to get accepted as true, a class of 'silent tool failures'
Method: rules called outcome contracts are mined automatically from task-disjoint clean execution traces or derived from public API schemas; when a tool result violates one, the original result is kept unchanged and a nonbinding receipt is appended naming the violated property and listing publicly available recovery tools, leaving the choice of what to do entirely to the agent
Result: on ToolMaze, completion rose from 10.9% to 28.1% averaged across four models from two provider families, and the gain replicated in a third provider's model (MiniMax M3); on tau-bench retail it improved by 14.0 and 12.0 percentage points across two difficulty tiers
Verification: removing just the recovery-tool list from the receipt eliminated the gain, and restoring it brought the gain back, while adding more diagnostic detail or changing the timing of the warning made no detectable difference, showing the actionable tool list is what actually drives the effect
Limitation: on a suite built from a real published catalog of production incidents, detection accuracy dropped to 46% for failure types outside the mined vocabulary, though the system still delivered results and overall completion was unaffected

Figure 1: A raw interface leaves detection, localization, and recovery with the agent. Detect and advise instead flags a contract violation and appends an advisory receipt carrying a diagnostic witness and a list of available recovery tools. It neither removes actions nor selects a repair. Controls (Section 5) identify the recovery-tool list, not the property-specific witness, as the active content in our tested sample.

Table 1: Conceptual comparison of recovery interfaces. Shaded cells mark where a strategy differs from the Outcome Monitor row, which serves as the reference; the shading locates design differences and does not rank them. The table characterizes mechanisms, not measured superiority.
Strategy	Trigger	Grounding	Localized witness	Restricts actions	Extra tool reads
Generic caution	learned detector event	external invariant	no	no	no
Self-critique	model’s own judgment	internal	uncertain	no	optional
Always verify	every advertised read	none (fixed policy)	only after comparison	no	yes
Hard guard	monitor-detected event	external rule	yes	yes	no
Outcome Monitor	learned detector event	external invariant	yes	no	no

Table 2: Frozen paired ToolMaze completion. The four-model primary aggregate clusters shared tasks; MiniMax M3 is a separately frozen third-family replication and is not pooled. Δ is in percentage points; W/L counts discordant pairs. All models run with extended reasoning disabled. Primary task-cluster sign-flip p<.00001; 95% bootstrap interval [11.25,23.44].
Family	Model	Base	Monitor	Δ	W/L
DeepSeek	V4 Flash	14/80	27/80	+16.25	15/2
DeepSeek	V4 Pro	13/80	23/80	+12.50	14/4
Qwen	3.7 Plus	3/80	15/80	+15.00	12/0
Qwen	3.7 Max	5/80	25/80	+25.00	21/1
Primary aggregate	35/320	90/320	+17.19	62/7
MiniMax	M3 replication	5/80	20/80	+18.75	17/2

Why it matters

As AI agents increasingly chain tool calls to automate real work, documented incidents show they can mistake corrupted or stale data for fact and produce confident but wrong outputs. This work shows a cheap, training-free, verifiable way to substantially reduce that failure mode without adding extra model calls.

Terms in this paper

Outcome Contract · a rule that a normal tool result must satisfy, automatically mined from clean execution traces or public API documentation
Outcome Monitors · the paper's system that checks tool results against outcome contracts and attaches a nonbinding warning when one is violated
Silent Tool Failure · a failure where a tool result is wrong but arrives in the expected, seemingly valid format, unlike an obvious error like a timeout
ToolMaze / tau-bench / AppWorld · benchmark environments used to test how well AI agents perform tasks by calling external tools
Recovery Affordance · an alternative tool or action the agent can actually choose to use when a problem is flagged

Original abstract (English)

When a tool call times out, the agent sees the failure and can route around it. A cached error page or negative price can instead arrive in the expected format and be consumed as fact. We introduce Outcome Monitors, which detect violations of outcome contracts mined from task-disjoint traces or derived from public schemas. On a violation, the monitor preserves the result and issues a nonbinding receipt naming the violated property and public recovery tools. In frozen, prespecified evaluations with injected failures, Outcome Monitors raise ToolMaze completion from 10.9% to 28.1% across four models in two provider families and replicate in a third. In tau-bench retail, completion improves by 14.0 and 12.0 points on two tiers. In separate ToolMaze controls, removing the recovery-tool list eliminates the measured gain and restoring it recovers the effect; diagnostic detail and timing produce no detectable differences. Gains concentrate where the fault blocks completion. On a suite transcribed from a published incident taxonomy, detection outside the mined vocabulary falls to 46%, though delivery continues and completion is unchanged. Recovery tools are the active receipt content in these controls; extending detection beyond the contract vocabulary remains open.

Authors · Sugam Panthi, Rabab Abdelfattah

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Sugam Panthi et al., arXiv:2608.19303, CC BY 4.0