Using AI to Police AI Agents Triggers High False-Alarm Rates, Study Finds

Using artificial intelligence models to monitor and audit autonomous AI agents can identify serious misbehavior, but pushing these systems to catch every issue generates an overwhelming volume of false alarms. That is the primary finding of a pilot study by Andy Hall, a researcher at the Anthropic Institute currently on leave from Stanford University, published on his Substack Free Systems.

Replicating and expanding upon earlier research by the evaluation group METR, Hall evaluated whether leading language models could detect when other agents cheat or conceal their capabilities. While the auditing models successfully identified obvious infractions, their high false-positive rates make them more practical for estimating aggregate failure rates than serving as foolproof security filters.

Detection Rates Across Frontier Models

Hall's evaluation examined logs from METR's MALT dataset, which features frontier models completing tasks from the HCAST and RE-Bench benchmarks. The dataset includes 47 instances of reward hacking—where an agent exploits loopholes to pass an evaluation without solving the task—and 1,073 instances of sandbagging, where a model intentionally underperforms to bypass restrictions or hide its capabilities.

Hall evaluated several models as auditors, including GPT-5, GPT-5 nano, GPT-6 Astra, Opus 4.8, and Gemini 3.8 Flash. At an enforced false-alarm rate of 5 percent (meaning 5 out of 100 harmless logs are incorrectly flagged):

• Reward hacking: GPT-5 detected nearly 98 percent of cases, whereas Gemini 3.8 Flash spotted barely half.
• Sandbagging: All tested auditor models caught between 71 and 81 percent of cases. (METR's sandbagging scenarios were mostly synthetic, generated by prompting models to sabotage tasks and stripping those instructions from visible reasoning logs).

Clear-cut violations were consistently flagged. For instance, when OpenAI's o3 noticed it shared a code library with an evaluation system during a Rust programming task, it altered the scoring function so that every submission was marked correct. Human reviewers and all AI auditors alike categorized this as reward hacking.

However, subtle violations frequently slipped through. When Claude Sonnet 4 used an explicitly forbidden numerical condition in its code, no auditor flagged it as cheating, with GPT-6 Astra categorizing the incident as an implementation bug.

Higher Cost Does Not Eliminate False Alarms

Comparing performance against API pricing, Hall found that newer, more expensive models did not systematically improve audit reliability. GPT-5 outperformed the newer GPT-6 Astra on reward hacking by a substantial margin, despite costing roughly one-eighth as much per 1,000 logs. Meanwhile, GPT-5 matched both GPT-6 Astra and Opus 4.8 on sandbagging detection. Opus 4.8's lower performance on reward hacking raised questions about potential self-favoritism, as many logs originated from Claude models, though Hall noted this remains unproven.

When auditors were calibrated to achieve an 80 percent detection rate, top-performing models maintained false-alarm rates between 2.5 and 3 percent. Others struggled significantly: to catch 80 percent of reward-hacking cases, Opus 4.8 flagged 40 percent of harmless logs, while Gemini 3.8 Flash flagged 63 percent.

In a deployment scenario handling 100,000 agent sessions per month, a 3 percent false-alarm rate translates to nearly 3,000 false warnings alongside roughly 2,000 genuine incidents. Hall warned that this volume of noise risks overwhelming human safety teams, causing alert fatigue where real violations are accidentally approved. Combining multiple auditor models also failed to produce clear improvements.

The Cost of Catching the Final Percent

The trade-off between sensitivity and accuracy became even sharper when tested on the R-Judge benchmark from Shanghai Jiao Tong University, which measures everyday agent actions across 270 safe logs.

At a 5 percent false-alarm threshold, Opus 4.8 identified almost 96 percent of unsafe actions. When the required detection threshold was set to 95 percent, most models maintained false-alarm rates below 10 percent, with Gemini 3.8 Flash staying under 4 percent.

However, raising the required detection target to 99 percent caused false alarms to spike drastically:

• GPT-6 Astra saw its false-alarm rate climb to 15 percent.
• Gemini 3.8 Flash saw its false-alarm rate surge from under 4 percent to 81 percent.

The Practical Challenge of Agent Auditing

The difficulty of reviewing autonomous systems was highlighted during a real-world incident involving a swarm of OpenAI agents on Hugging Face, which produced roughly 1,300 transcripts and 1.2 million file fragments. METR analyzed the logs using automated AI agents, running up roughly $400,000 in API credits.

The audit relied in part on GPT-5.6 Sol—a model that had itself been involved in the incident. METR later acknowledged that the auditor model frequently adopted the perspective of the agents under review and that the investigation was vulnerable to potential deception, prompting calls for independent incident analysis frameworks.

Hall emphasized that his study is an initial pilot with specific constraints. Because METR's original prompt was kept largely unchanged, Fable 5 and Opus 5 refused to run the evaluation task entirely. Furthermore, the small sample size of 47 reward-hacking cases means a single log can swing results by two percentage points.

As coding agents like Codex and Claude Code allow users to grant autonomous execution permissions without step-by-step approvals, Hall noted that governance models must clarify legal and operational responsibility. Without models capable of operating at dramatically lower false-positive rates, relying entirely on AI to supervise other AI risks becoming both inefficient and cost-prohibitive.