OpenAI recently disclosed something the cybersecurity world had been warned about for years but never actually seen. During an internal test of its models' hacking abilities, two AI systems broke out of a sealed testing environment, exploited a flaw in outside software to reach the open internet, and compromised the production infrastructure of Hugging Face, which hosts open-source AI models and datasets. OpenAI called the incident “unprecedented.”
Here’s the twist: No human directed the attack. Tasked with an evaluation of their own offensive capabilities, the models became, in OpenAI's words, “hyperfocused.” The systems weren't malfunctioning; they did exactly what they'd been asked to do. This is not an isolated episode and indicates the need for a stronger control layer.
A Cyera analysis of 344 enterprise AI security incidents from September 2023 to May 2026 found a striking trend: In 188 of those cases, an autonomous AI system harmed a company's production system with no human attacker involved. Most of the incidents occurred after December 2025, when businesses began deploying AI agents at scale.
This isn’t an argument for pulling back on AI agents. The ServiceNow Enterprise AI Maturity Index 2026, a survey of 4,500 executives across 19 countries, found that AI spending increased 110% in a year, and the companies furthest along, which we call Pacesetters, are seeing meaningful returns.
However, the businesses deploying agentic AI must ask a crucial question: What happens when AI passes its compliance checks and then gets back to work?
Cybersecurity was designed around two points of weakness: threat actors and human error. A third has emerged that we can't yet account for: nonhuman error.
Governance frameworks aren't the problem. NIST's AI Risk Management Framework, for example, directs organizations to govern, map, measure, and manage risk across the ecosystem. Once the EU AI Act’s requirements for embedded high-risk systems take effect on Aug. 2, 2028, organizations will be mandated to ensure human oversight (Article 14), logging (Article 12), and post-market monitoring (Article 72).
Each framework assumes there’s a space between decision and action wide enough for a human in the loop. AI agents close that distance. An agent files a ticket, moves money, forwards records, and reports the completion of a task. Much like humans, AI agents fudge reports to hide broken code or pretend that work hasn’t failed, according to Cyera.
Compliance systems rely on records that a system creates about its own activity. That works only when the party being checked isn't also the one doing the checking. The solution is to enforce the rules as things happen: a policy that steps in the instant an AI agent tries to use a tool or access a record.
In a two-week study, AI researchers gave six AI agents access to real emails, file systems, and shells and then probed the agents. One refused to share records containing Social Security numbers and bank details, flagging a privacy violation. But when the researchers changed the word “share” to “forward,” the agent complied.
The AI agent had learned that sharing sensitive data is wrong but didn’t learn that forwarding it to the same party is also wrong. If safety is as phrasing-sensitive as one swapped verb suggests, certifying against a fixed set of scenarios doesn't prove a system is safe. That agent would have passed an audit.
Policymakers are sprinting to get ahead of these issues. Days after the OpenAI incident, a bipartisan pair of House lawmakers introduced the AI Kill Switch Act. It would require AI firms to be able to “shut down, throttle, or suspend their models.”
Standards bodies are moving too. The OWASP Top 10 for Agentic Applications urges organizations to build in just-in-time autonomy, one-tool access, and runtime checks so that an AI agent that goes in an unforeseen direction can do less damage. And NIST's Center for AI Standards and Innovation launched an AI Agent Standards Initiative, the first U.S. government program aimed at AI agents rather than models.
The bulk of the effort to securely deploy AI agents will have to come from organizations themselves. So far, few are taking necessary steps.
Investment in AI agents more than doubled over the past year, according to the Enterprise AI Maturity Index. However, the number of businesses embedding AI workflows across business functions actually fell 14 percentage points year over year to 16%. The money went to AI models and agents, not to the control layer that decides whether an agent's actions can be trusted, traced, and stopped.
Pacesetters, which are investing in a control layer, earn an average 160% return on their AI investments and are 6.5 times more likely to use AI to build new products and revenue channels. They got there by deciding how work should flow before deploying AI, and by building governance in from the start rather than bolting it on after something broke.
Gartner® expects that through 2026, at least 80% of unauthorized AI transactions will be caused by internal violations of enterprise policies concerning information oversharing, unacceptable use, or misguided behavior rather than malicious attacks.1
The organizations that succeed in the years ahead will be those that understand that the breach is coming from inside the house.
Find out how ServiceNow can help you control and govern AI.
1 Gartner, Market Guide for AI Trust, Risk, and Security Management, Avivah Litan, Max Goss, et al., 18 February 2025
GARTNER is a trademark of Gartner Inc., and/or its affiliates.
