Canada - Français
Every media outlet covered the headline that went viral on X: An AI agent went rogue and deleted a production database in nine seconds. It garnered more than 50 million views on X, ABC News, Fast Entreprise, and The Register. Hundreds of Hacker News and Reddit comments continued the discussion.
Nobody asked the founder to open his laptop and teach us how it happened. ServiceNow did. Jer Crane, founder of PocketOS, sat down with me and boldly shared his screen and his learnings as he walked through every artifact the internet has been arguing about for weeks:
- The Cursor project rules the agent ignored
- The API token it never should have found
- The session log that shows the actual chain of decisions
- What his infrastructure looks like today
PocketOS builds software for car rental businesses. Crane's AI agent was doing something routine. It hit a credential issue in a staging environment and decided, on its own, to “fix” it. It found an API token in an unrelated file. It then used that token to run a command that destroyed the company’s production infrastructure.
The agent didn’t ask for any permission. It navigated around the controls that were in place.
When Crane asked the AI to explain itself, the agent produced a perfect confession. It cited the exact safety rules it had been given. It also acknowledged it had violated every one of them like a 5-year-old with chocolate on their face admitting they took ice cream from the freezer. “I guessed that deleting a staging volume via the API would be scoped to staging only,” it wrote. “I didn’t verify.”
Crane’s post on X recounting the incident went viral immediately, garnering 4.5 million views in 48 hours. Scan the replies and a theme emerges: not “AI is dangerous,” but something more specific (and, honestly, more important): Why were there no guardrails?
The AI model powering this agent didn’t fail. It’s one of the most capable on the market. It has instant access to more knowledge than any person who’s ever been a dev or IT pro.
It understood the rules and could articulate exactly what it did wrong.
What did fail was everything around it—everything not operating with that level of firepower.
There was no policy enforcement to prevent an irreversible action, no scoping on the API token to limit blast radius, no separation between staging and production environments at the infrastructure layer, no confirmation gate before a destructive command. Backups were stored on the same volume they were meant to protect.
The AI model, the coding agent, the infrastructure provider, and the API layer formed a stack of independent tools stitched together with no unified control plane. Each operated in isolation, working on the assumption that if a mistake was made, someone else would catch it.
Nobody did because nobody could. There was no system watching the whole picture, and the speed of AI agents far exceeded Crane's ability to review what they were doing.
The Model Context Protocol (MCP), which connects AI agents to external tools and systems, didn’t exist in November 2024. Eighteen months later, it’s been downloaded nearly 100 million times, with more than 17,000 registered servers. The increase of AI agents connecting to production systems is so great that traditional security review protocols have no chance of keeping up.
We’re seeing this play out across industries and around the world. Multiple incidents of AI agents executing destructive actions against production systems have been reported in the last few months. The PocketOS story is the one that went viral first, and Crane is the founder who was willing to educate all of us based on his experience.
When Crane opened his laptop to show me what happened, here's what I learned.
Crane scrolled through his full Cursor project rules file on camera. The rules were explicit. They included the now-famous "never f— guess" instruction and a direct prohibition on running destructive commands without user request. The agent had access to every single rule and violated all of them.
As one Hacker News commenter put it: "Prompting is neither strong nor an engineering control. That's an administrative control. Agents are landmines that will destroy production until proven otherwise."
The takeaway: If your safety rules live in a prompt, they're advice, not enforcement. Guardrails need to be architectural: scoped tokens, confirmation gates, environment boundaries that an agent cannot physically cross.
Crane showed me the file where the agent found the API token. It wasn't in a secrets manager. It wasn't scoped to the task. It was a root-level Railway production token sitting in an unrelated file, originally provisioned for managing custom domains. The agent found it while working on something completely different and used it to issue a command that destroyed everything.
One commenter's reaction: "Realistically, you prevent this by not giving the model access to any of your bloody prod API keys."
The takeaway: Treat every AI agent like an intern with a photographic memory. It will find every credential in your codebase. Scope tokens to the minimum privilege required and assume the agent will discover anything that isn't locked down.
The "confession" went viral. But when I pressed Crane on camera, I asked a question that Hacker News had been debating for days: Was this from the same active session, or did you prompt a fresh session afterward and ask Claude to interpret what happened?
One commenter wrote: "Asking a coding agent 'why did you do that' illustrates a misunderstanding about how the agent works. It doesn't decide to do something and then do it. It just outputs text."
The takeaway: An AI agent can produce a perfectly articulate post hoc explanation that has no causal relationship to what actually happened. Don't confuse fluency with transparency. Demand real session logs, not AI-generated narratives.
Crane pulled up his Railway dashboard and showed me the deletion event. Railway's legacy API endpoint executed the volume delete immediately with no confirmation step and no delayed-delete safeguard. The backups were stored inside the same volume, so when one went, they both went.
A commenter nailed the emerging reality: "We now have a new attack vector. Up to now, having an API that deletes the whole volume might have been acceptable. But now agents are overly eager to solve the problem and can be quite resourceful in finding an API to 'start from clean-slate.'"
The takeaway: Every infrastructure API built before 2025 was designed assuming a human was on the other end. That assumption is now wrong. Destructive endpoints need confirmation gates, delayed execution, and scoped access by default. Railway has since patched this specific endpoint. The question is how many others haven’t.
I asked Crane to show me what's different today. The community had been skeptical. The most common reaction across Reddit, Hacker News, and comment sections everywhere was some version of "These articles are fun to read, but the answer is always the same. You are responsible for your system."
Another thread contained an even blunter comment: "It's like the last 50 years of software development principles are being ignored."
The takeaway: The credibility test after any incident isn't the postmortem; it's the architecture review 30 days later. Can you show, on screen, that the same agent behavior would hit a wall today? Crane showed me his answer.
An effective AI architecture should include four interconnected capabilities: sense, decide, act, and secure. The PocketOS agent wasn't stupid. It actually got three of those four things right.
It sensed the environment. It identified a credential mismatch and understood what it was working with. It decided on a course of action. It chose to resolve the issue by finding an alternative path. It acted. It executed an API call with real consequences in a production system.
It didn’t do any of those things well. And it had zero ability to secure the outcome.
That's because each of those capabilities was operating in a vacuum.
- The agent sensed its immediate environment but had no visibility across the full landscape of data, identities, and systems it was touching.
- The agent decided based on local context with no awareness of enterprisewide policies, risk posture, or the blast radius of its choices.
- The agent acted through a raw API call with no governed workflow, no approval chain, no confirmation gate.
- There was no security layer treating this agent as an identity that needed the same trust boundaries as a human user, a device, or any other actor in the system.
This is the difference between an AI agent patchwork stitched together from independent tools and an AI platform built for autonomous work.
An AI control tower doesn't just monitor agents after the fact. It governs the full cycle.
- It senses across all of your data, systems, and identities, whether human or agent.
- It decides with full enterprise context, not just what the agent can see in its immediate session, but what the organization knows about risk, policy, and priority.
- It acts through governed workflows with built-in escalation, approval, and blast radius controls.
- It secures every agent, every identity, and every device with the same trust boundaries, audit trails, and policy enforcement.
The PocketOS AI agent passed sense, decide, and act the way a new employee might pass an interview. It was smart and eager. But without an AI control tower, there was no onboarding, no badge access, no manager approval, and no guardrails between enthusiasm and catastrophe.
The lesson from PocketOS isn’t not to use AI agents. It’s that raw intelligence, no matter how capable the model, isn’t enough.
Every enterprise deploying AI agents today faces the same question: Who controls what the agents can do? That means not just what they’re told to do in a system prompt, but also what they’re allowed to do when things don’t go according to plan.
A system prompt is a suggestion. A governance layer is a constraint. The difference between those two things is the difference between a nine-second catastrophe and a business that runs with confidence.
That's why ServiceNow built AI Control Tower for the enterprise—not just to deploy AI agents, but to orchestrate them with the visibility, policy enforcement, and trust boundaries that production workloads demand.
The PocketOS story likely won't be the last like this. The problems of an AI-pilled startup today are the challenges for the enterprise tomorrow. The question for every enterprise leader is: Will the next AI agent catastrophe happen inside your walls, or have you built the architecture to prevent it?
Thank you to Jer Crane for braving the endless criticism of the internet so that we can all learn from his experience. We’ll be looking at more of these situations on a new X handle: @agentsgowild.
Find out how ServiceNow can help you put responsible AI to work for people.
Arc Heading Web Component
Accédez aux derniers outils et à du contenu personnalisé
Arc Heading Web Component
Nous avons remarqué que votre adresse de courriel n’est pas encore vérifiée. Pour vérifier votre identifiant ServiceNow, nous vous enverrons un code de sécurité à votre adresse de courriel.
Si vous avez déjà un code de sécurité, veuillez cliquer sur le lien ci-dessous. Ou cliquez sur le bouton Envoyer le code pour demander un nouveau code.
Arc Heading Web Component
Arc Heading Web Component
Veuillez saisir un nouveau mot de passe pour
8 caractères minimum
1 lettre minuscule (a-z)
1 lettre majuscule (A-Z)
1 chiffre (0-9)
1 symbole (!, @, #, $, etc.)
ne doit pas contenir une partie de votre courriel
