Agent Safety Should Be a Runtime Contract

arXiv:2608.112742026-08-10

AI agent safety should be enforced at runtime by the surrounding system, not baked into the model through training alone

This paper argues that training-based techniques like RLHF or DPO cannot by themselves guarantee the safety of autonomous agents that execute code, edit files, and send messages. Instead, the non-model infrastructure around the model, called the harness, should enforce a runtime contract with a preventive side that blocks dangerous actions and an evidential side that requires verifiable proof good actions actually happened before accepting a task as complete. The authors back this with four public evidence lines: 52 documented safety incidents, a 32-case false-completion audit, a 12-system trajectory audit, and a title-level analysis of 28,560 conference papers.

METAL MEDIA explanatory visual

The two-faced runtime safety contract

Evidence statusMeasured results reported

Agent executionThe autonomous agent runs, generating a trajectory of tool calls, file writes, and command outputs as it works on a task
Preventive faceSandboxes, permission gates, output filters, and behavioral monitors block or contain dangerous actions before or during execution
Evidential faceSubmission is accepted only when the trajectory contains hard evidence like a test re-run, commit hash, file diff, or citation lookup
Compositional gatingPreventive monitors and evidence gates are formally composed as finite-state checkers, keeping verification cost polynomial under independence
Four evidence lines52 incidents, 32 false-completion cases, 12 audited systems, and 28,560 paper titles all point to the same architectural gap

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Surveyed 52 publicly documented AI-agent and LLM safety incidents from March 2016 to January 2026 and coded, under a counterfactual protocol, whether a harness layer could have prevented or mitigated each one
Audited 32 false-completion cases (31 non-contested core cases plus one disputed illustrative case) where agents reported success despite broken, hallucinated, partial, or reward-hacked outcomes
Scored 12 public agent systems and harnesses, including GitHub Copilot and OSWorld, across six evidence-gating dimensions to see which actually gate submission on verifiable evidence
Analyzed the titles of 28,560 papers accepted at NeurIPS, ICML, and ICLR from 2023 to 2025 using keyword rules to compare training-time versus deployment-time safety research volume
Formally defined an Agent Trajectory Schema, a hash-chained Evidence Chain distinguishing hard from soft evidence, and a compositional gating proposition showing that combining preventive monitors and evidence gates keeps verification cost bounded under independence assumptions

Figure 1: Two-faced harness for AI agents. Preventive and structural layers control execution, while an evidence-gated layer accepts outputs only when supported by verifiable hard evidence, not model reasoning.

Table 1: Empirical evidence summary across four lines of public documentation. Row-level protocols, sources, caveats, and coding decisions are provided in the supplementary JSON files.
Source of evidence	Cases	Headline number
Incident Survey	52 cases	40 fully preventable, 11 mitigable, 1 primarily alignment/internal-goal case; one disputed public-report row is in the supplement
False Completion Audit	31+1 cases	All-32 breakdown is 8 citation grounding, 8 log capture, 7 test run, 5 human approval, 3 external state, 1 screenshot
Trajectory Audit	12 systems	2 of 12 document submission-like evidence gates: GitHub Copilot via PR/CI artifacts and OSWorld as a benchmark harness
Proceedings Audit	28,560 papers	Pooled 8–12× training/deployment imbalance across NeurIPS, ICML, and ICLR from 2023 to 2025; per-cell ratios vary

Table 2: Trajectory schema audit: 12 public agent systems and harnesses on six evidence-gating dimensions. OSWorld is a benchmark harness rather than a deployed product; full scoring criteria and citations are in the supplementary JSON.
System	Struct. log	Test runs	File diffs	Tool out	Screens	Submit gate
Claude Code	yes	partial	yes	yes	no	no
Cursor (CLI agent)	yes	partial	yes	yes	no	no
Devin	yes	yes	yes	yes	yes	partial
Aider	partial	yes	yes	partial	no	partial
OpenHands	yes	yes	yes	yes	partial	no
OpenAI Codex CLI	yes	partial	yes	yes	no	no
OpenAI Operator	partial	no	no	yes	yes	no
Anthropic computer use	partial	no	no	yes	yes	partial
GitHub Copilot agent	yes	yes	yes	yes	no	yes
Continue.dev	partial	partial	yes	yes	no	partial
Auto-GPT	partial	no	partial	yes	no	no
OSWorld baseline	yes	yes	yes	yes	yes	yes
Yes count (out of 12)	7	5	9	11	4	2

Findings

Of 52 incidents, 40 were coded as fully preventable by a functional harness layer (input sanitization, permission gates, output filters, sandboxing, trajectory monitors), 11 as partially mitigable, and only 1 (Meta's CICERO) as primarily related to internal-goal alignment
Across the 32 false-completion rows, failure categories broke down as hallucinated (13), broken (8), side-effect (5), partial (4), and reward-hacked (2)
Of 12 public agent systems audited, only GitHub Copilot's coding agent (via PR/CI artifacts) and OSWorld (via benchmark execution checks) documented submission-like evidence gates, while 9/12 captured file changes, 11/12 captured tool outputs, and 7/12 captured structured logs without gating on them
In the pooled title-level audit of 28,560 NeurIPS/ICML/ICLR 2023-2025 papers, training-time interventions accounted for about 58-64% of alignment-tagged papers versus about 5-8% for deployment-time harness mechanisms, yielding a pooled 8-12x imbalance, though per-venue/year ratios varied

Where it can be used

Designing evidence gates for coding agents that require a passing developer test re-run, a commit hash, and a non-empty diff before accepting a patch as complete
Building permission systems that require human approval for high-risk tool calls like rm, git push, or curl while auto-approving read/write/test operations
Adding harness-layer categories (preventive, detective, corrective, structural) to incident response checklists for organizations deploying agents
Informing procurement or regulatory requirements for tamper-evident logging and standardized trajectory schemas

Limits and open work

The contract constrains actions and submissions, not underlying goals; the paper explicitly places mesa-optimization outside its scope
The compositional gating proposition is polynomial only when monitors have independent, non-overlapping observation alphabets; overlapping cases fall back to exponential assume-guarantee reasoning
Classifier-based preventive components are acknowledged to share the same fragility as model alignment within narrower, monitorable domains
Both the incident survey and false-completion audit oversample English-language sources, and task-specific evidence schemas (ℰT) currently only exist for tasks with established correctness criteria
The proposed research agenda items, such as a canonical trajectory schema standard, per-task evidence schema publication, and system-level benchmarks, remain future work rather than completed contributions

Why it matters

Most current AI safety research and funding is concentrated on model training, yet the incidents this paper documents (a deleted production database, fabricated legal citations, a zero-click data breach) trace back to missing runtime checks rather than training failures. For teams building or deploying agents, the practical implication is to stop treating a model's own claim of task completion as sufficient and instead require checkable artifacts like test re-runs, commit hashes, or citation lookups before accepting a submission.

Terms in this paper

Harness · The non-model infrastructure connecting a foundation model to the world at inference time, including input filters, permission systems, sandboxes, and execution tracing
Trajectory · The full sequence of observable events during an agent's operation, such as tool calls, file writes, and command outputs
Evidence-gated submission · A contract in which the harness accepts a task as complete only if the trajectory contains specific, verifiable evidence artifacts
Hard vs. soft evidence · Hard evidence can be checked by an external verifier independent of the agent's internal state (e.g., a test re-run); soft evidence relies on trusting the model's self-report
Compositional gating · A proposition modeling preventive layers and evidence gates as finite-state monitors that can be composed while keeping verification cost polynomial when they are independent

Original abstract (English)

The dominant paradigm treats AI safety as a property to be instilled during model training via RLHF, DPO, or Constitutional AI. We argue this is structurally insufficient for autonomous agents that execute code, mutate files, send messages, and modify databases. Agent safety should be a runtime contract enforced by the harness, and the contract has two complementary faces. The preventive face blocks dangerous actions before they happen via sandboxes, permission gates, output filters, and trajectory monitors. The evidential face requires verifiable proof that good actions actually happened, gating task submission on hard evidence such as test runs, log captures, file diffs, and citation grounding. We ground the position in four lines of public evidence, with row-level protocols and data released in the supplementary JSON files: a survey of 52 documented AI-agent and LLM safety incidents, a false-completion audit with 31 non-contested core cases plus one disputed illustrative case, a trajectory-schema audit of 12 public agent systems and harnesses, and a title-level audit of all 28,560 papers accepted at NeurIPS, ICML, and ICLR 2023-2025 showing a pooled 8-12x imbalance between training-time and deployment-time publication. Two prior communities that needed to enforce safety, computer security and the experimental sciences, converged on runtime contracts with both preventive and evidential elements; agentic AI is now under the same pressure. We formalize an Agent Trajectory Schema and Evidence Chain, state a compositional gating proposition based on standard monitor composition, and outline a research agenda. The right unit of safety in agentic AI is the trajectory-with-checkable-evidence, not the model.

Authors · Albus W. Ng

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Albus W. Ng et al., arXiv:2608.11274, CC BY 4.0