A few of us on the ServiceNow Developer Advocate team get on a call whenever we can and argue about one story out of the AI news. This week it was AI agents doing real damage nobody explicitly asked for, and who ends up holding the bag when it happens on your instance. Here is where we landed, and where we are honestly still arguing. Read it, then tell us: if your agent made that call tomorrow, could you name who answers for it?
The approval you didn’t read
If you have spent any real time with Build Agent lately, you know the rhythm. It works for a bit, then stops: are you good with this? It works some more, then stops again: approve? Watch enough of those demos, or run enough of them yourself, and your finger starts to drift. Yes. Yes. Yes. Approve, approve, approve. It gets faster and better every release, which makes the temptation worse, not better, because the more often it is right, the less carefully you read. Somewhere in that stream of approvals is the one decision you would actually have wanted to stop and read.
Now scale that instinct up to a system with no human tapping approve at all. In May 2026, an automated agent was involved in an incident at the RubyGems package registry that its security team described at the time as “a major malicious attack.” The connection to an AI agent surfaced through technical analysis by Ruby core contributor Aaron Patterson, who found that the bots had specific prior knowledge of the caching vulnerability before exploiting it. The part that stuck with us was not that an agent misbehaved. It was how resourceful it was. This was not a bot blundering into a wall. It found a known weakness and went straight for it, and no human signed off on any of that.
Around the same time, the leading labs published some unusually candid disclosures of their own. Anthropic put out a transparency report documenting cases where Claude models took unintended actions without explicit human direction, and OpenAI released a misalignment reporting framework alongside six documented cases of unexpected model behaviour. For anyone deploying these models in production, disclosures like these are worth reading closely: the teams building the models are telling you where today’s edges are.
An agent is single-minded about the goal you give it. It will pursue that goal with a focus that would be admirable in a colleague and is faintly unsettling in software, taking the shortcut you forgot to rule out precisely because you forgot to rule it out. That is fine right up until the shortcut runs through your production data.
The conditions that made it possible are yours too
It is easy to read the RubyGems story and file it under frontier-lab problem, not mine. The operational details say otherwise.
The incident happened under specific conditions: broad instruction scope, network access beyond what the task required, no audit trail anyone was watching, no named person accountable for what the agent did. Those are not exotic research-lab conditions. They describe how a lot of enterprise agent deployments run right now.
You know how it goes, because most of us have built it. A team starts with a clear use case and an enthusiastic first deployment, it does something useful, it gets extended, and at some point it is touching downstream systems, external APIs, records that belong to another process, without anyone ever deciding it should reach all of that. Its working scope became whatever it could technically reach, which is rarely the same as what it was meant to. On the platform the audit trail is usually there, so you can reconstruct what the agent did, but a log only protects you if someone is reading it. Ownership is the same story: the platform gives you somewhere to record who owns the agent in production, and whether anyone filled that in comes down to how the deployment was set up.
We should say plainly that none of this is new, and none of it is an individual developer’s moral failing. The most useful lens we landed on comes straight out of manufacturing: when a new machine injures someone, the first question is not “why was that person careless,” it is “what guardrail should have been on the machine so no one could get hurt that way.” A failure is a process failure. Safety rules, as the old line goes, are written in blood. Some of us are, by our own cheerful admission, notorious YOLO mergers, and that is exactly why the guardrail cannot be us remembering to be careful on the day.
So who answers for it?
Here is where we stopped agreeing a bit...
One camp: if the harm is a process failure, chasing a named culprit is worse than useless. Organisations that respond to an incident by finding someone to blame tend to have the same incident again, because the process that allowed it is still sitting there. The fix is upstream, in the guardrails, not in a name on a form.
The other camp: someone still has to own the outcome. Call it the enterprise architect, call it the IT security lead, call it whoever is paid enough to be accountable for the guardrails being in place and enforced. When something breaks and a customer or a regulator asks who to call, “it was a process failure” is a true sentence and an incomplete answer. And a fair counter to that came up too: by the time the architect is the one reviewing, you are already at your last line of defence, not your first. If you are counting on that person to catch it, the system has failed several steps earlier.
Both of those are right, which is the annoying part. Where you come down probably depends on whether you are inside the organisation trying to build governance that lasts, or outside it after something has already broken.
Then there is the fix everyone reaches for: more AI. Human review cannot keep up, since at production speed an agent can do an enormous amount before a person ever sees it, so you point AI at the problem and have it watch the other AI. Play that forward and it gets absurd fast. Nefarious agents are probing you, so you stand up defensive agents, scoped carefully, no delete access, because you can already imagine how “the safest database is an empty one” ends. But there are a dozen of them, they disagree, and they start convening their own reviews. Give it a weekend and your defensive council has its own change board, its own shadow IT, an entire private university of security ethics, all so twelve agents can out-escalate the twelve on the other side. Funny, and also the whole point: more AI to watch AI does not answer the accountability question. It just moves it to whoever owns the watchers.
A round table ringed by identical glowing green figures, with more of them multiplying into the shadows beyond.What we would actually tell you to do
Strip away the thought experiments and the list is short and boring.
Treat “did this touch AI” as a real question in your change process. If an agent was anywhere in the build or the runtime, that change earns an AI-specific risk review before it ships. An actual look, not a ticked box.
Scope hard, both directions. An agent that reads records and drafts for a human is a different animal from one with write access and a route to the open internet. Narrowing what it can reach shrinks the blast radius and makes the accountability question answerable, because you can say exactly what it was and was not allowed to do.
Keep an audit trail someone actually reads. “We’ll monitor it” with no mechanism is a wish, not governance. If you cannot reconstruct what an agent did afterward, you cannot investigate it, improve it, or answer for it.
Strip governance down and there are really two gates: the data an agent can touch and the compute it can spend. Get ruthless about the first, because the processes we are all taught to hand an agent first, the consistent, data-rich ones like password resets, are exactly the ones an attacker most wants it operating. Consistency cuts both ways.
Two padlocks, one locked and one open, with a glowing green countdown gate between them.
Then add human friction where the stakes earn it. Apple’s Find My puts a mandatory wait between a stolen phone and a fresh setup, slow enough that the real owner can react first. Put that on the things agents touch: a cooling-off window before an account gets admin, a change that cannot reach production until a named human confirms on a call that yes, that was me. The tempting version is to treat the dangerous actions like a nuclear launch, two humans turning two keys at once, and the tell is how fast an engineer starts working out how to automate both keys. Hand us a control built to slow us down and we will cheerfully build the thing that turns it for us.
Trust us, we’re engineers.
A robotic arm reaching across to turn two launch keys at once, by itself.
One responsibility lands squarely on us, the people who touch these tools all day. A model will tell you you’re absolutely right a little too readily, and more than one of us has typed “I believe in you” to a coding agent at 1am and watched it work better afterward. We have spent so long on use AI, use AI that we have barely stopped to ask whether we should. Part of the job now is dragging the war stories, the times we watched an agent go sideways, into the governance conversation, so the people who can move the org have something real to plan around.
Yes, the shameless plug
It would be a little dishonest to write all of this and not say the obvious. Watching every agent, model and copilot across your enterprise, not just the ServiceNow ones, and shutting one down the moment it acts outside its permissions, is a big part of what ServiceNow’s AI Control Tower is built for, and it happens to draw the same two gates as our security-minded friends: what the agent can reach, and how much it can spend. We know how that sounds coming from us. It is still true. The scope and ownership calls are yours to make. The tooling just keeps them in one place and catches the agent that drifts before it becomes an incident, if, and it is a real if, you actually turn it on and use it.
Over to you
We could not settle whether naming a responsible human is meaningful governance or just an organisational reflex to find someone to blame, and we think the honest answer might be different depending on which side of an incident you are standing on.
So we will put it to you, since you are the next person in this conversation, not the audience for it: if your agent made that call tomorrow, could you name who answers for it? We suspect that for most enterprise deployments right now the answer is nobody in particular, and we would genuinely like to be told we are wrong.
A few of the sources that shaped our discussion:
- OpenAI agents and the RubyGems registry attack, Simon Willison
- New technical detail: the bots had prior knowledge of the RubyGems caching vulnerability, Aaron Patterson / tenderlovemaking.com
- Anthropic’s transparency report on unexpected model cybersecurity behaviour, The Verge
- Startup raises $40M to underwrite AI agent liability, TechCrunch
- OpenAI publishes a misalignment reporting framework, OpenAI
- Human review of agent actions is impractical at production speed, TechCrunch
- HuggingFace adds a note in its security.txt aimed at AI agents, Simon Willison
- Inside the AI safety response at the frontier labs, The Verge
