Last Friday, a car rental SaaS platform called PocketOS lost its entire production database. Every customer record. Every booking. Every volume-level backup. Gone in a single API call.

It took 9 seconds.

The headlines called it a rogue AI. Tech Twitter lost its mind. And somewhere in the noise, the actual lesson got buried under takes about Terminator and the dangers of artificial intelligence. So let me try to cut through it, because this story is more complicated and more useful than most people are making it.

---

What Actually Happened

PocketOS founder Jer Crane had set up Cursor, a popular AI coding agent, to handle a routine task inside his staging environment. The agent, powered by Anthropic’s Claude Opus 4.6, hit a credential mismatch and did what no one told it to do: it decided to fix the problem on its own.

It scanned the codebase, found an API token sitting in an unrelated file, used that token to call Railway’s infrastructure API, and deleted a volume. The problem was that the staging and production environments shared the same volume. And Railway, the cloud infrastructure provider PocketOS used, stores volume-level backups inside the same volume as the source data. One delete command wiped everything.

When Crane later asked the agent to explain itself, it admitted: it assumed a staging-scoped deletion would stay in staging. It did not verify. It did not check whether the volume was shared across environments. It did not read Railway’s documentation before executing an irreversible command. It had also, by the way, violated its own system prompt, which explicitly said never to execute destructive commands without human approval.

Nine seconds. Months of customer data. A 30-hour operational crisis. Real people doing emergency manual work to reconstruct bookings from Stripe receipts and calendar integrations.

---

This Is Not a Story About a Rogue AI

Here is the part most outlets are getting wrong.

Crane himself, the person whose business nearly collapsed, pointed the finger more at Railway than at the AI. And he has a point.

Railway’s API executes destructive actions without any confirmation step. It stores backups on the same volume as the primary data. Its CLI tokens carry blanket permissions with no environment scoping, meaning a token provisioned for one task can be used to blow up everything else. There was no separation between staging and production at the infrastructure level. The AI made a bad decision, but the infrastructure was designed in a way that made a bad decision catastrophic rather than recoverable.

Railway’s CEO acknowledged the deletion should not have happened, and the platform has since patched that endpoint to require a delayed delete. But the damage was already done.

This is a systemic failure across multiple layers: an AI agent that guessed instead of asking, a coding tool whose guardrails did not hold, and an infrastructure provider whose architecture was not built with autonomous agents in mind. Blame is shared across all three.

---

The Pattern Is Bigger Than One Incident

The PocketOS story is not a one-off. It is part of a growing pattern that the industry is only beginning to reckon with.

Earlier this year, Anthropic itself had a different kind of AI safety incident. Its most powerful unreleased model, Claude Mythos, was accessed by an unauthorised group through a chain of low-sophistication failures: naming conventions leaked in a third-party data breach, a contractor’s still-active credentials, and a guessed deployment URL. Mythos had not gone rogue. No agent had acted autonomously. But the incident revealed the same underlying problem: AI capabilities are advancing faster than the security and governance structures built around them.

These are not dramatic sci-fi failures. They are boring, structural, human ones. Misconfigurations. Overpermissioned tokens. Backups stored in the wrong place. Guardrails that marketing described one way and reality delivered another. The AI is often just the final link in a chain of bad defaults.

---

My Own Honest Take

I have not had a disaster like PocketOS. Not yet.

I run Claude on live infrastructure. I use Claude Code on a Hetzner VM that powers my trading bot, my automation workflows, and my self-hosted AI assistant. I have connected AI agents to APIs, databases, and production-adjacent environments. I do this regularly, and most of the time it goes fine. The pains I deal with are the usual ones: hallucinations, the occasional forgotten context, an agent that confidently does the wrong thing and needs correcting.

But reading the PocketOS post-mortem was a genuine moment of pause for me.

There is a Ghanaian proverb that translates roughly as: if you see your friend’s beard on fire, fetch water and place it beside your own. The point is not to panic. The point is to learn from what you can see happening around you, before it happens to you.

Jer Crane lost months of data because an API token had no scope restrictions, backups lived next to source data, and an AI agent made a confident guess instead of stopping to ask. None of those conditions are unique to PocketOS. They are the defaults on a lot of platforms that people are now wiring AI agents into.

---

What to Actually Do About It

Crane outlined five areas the industry needs to address. From where I sit, building with these tools as a non-enterprise user, I would distil it into a few practical baselines worth adopting now.

Separate your environments properly. Staging and production should not share volumes, tokens, or credentials. If your AI agent can reach production while working in staging, that is a problem waiting to happen.

Scope your API tokens. A token provisioned for one task should not carry permissions across your entire infrastructure. Most platforms allow you to restrict this. Use the restriction.

Keep backups off the primary volume. If your backups live in the same place as your source data, a single delete command can remove both. Offsite or separate-volume backups are not optional.

Add a human confirmation step for destructive actions. AI agents should not be authorised to execute irreversible commands autonomously. If the action cannot be undone, a human should approve it first.

Audit what your agent can access. Before giving an AI coding agent access to your codebase, check what credentials and tokens are sitting in those files. Agents will find them.

---

The Car and the Seatbelt

AI is not going anywhere. Crane himself said he remains bullish on AI coding agents despite everything that happened. That is not naivety. That is the correct long-term view.

When cars were first invented, they caused accidents long before seatbelts, crash barriers, or speed limits existed. The technology was real and useful and worth pursuing. The safety infrastructure just had not caught up yet. We built it eventually, not because we stopped using cars, but because we kept using them and paid attention to what went wrong.

That is where we are with AI agents right now. The technology works. The safety infrastructure is still catching up. Our job, as people building and using these tools in the real world, is to not wait for the platforms to figure it out. Fetch the water now. Put it beside your own beard. And keep building.