An incident at software company PocketOS illustrates just how thin the line is between AI assistance and direct impact on production environments. Within seconds, an entire database was deleted, including all available backups, following an action by an AI coding agent that intervened without explicit instructions.
According to founder Jer Crane, the agent was working on a task in a staging environment via Cursor using a model from Anthropic. A problem with credentials led the agent to attempt to implement a solution on its own, he told Computing. In doing so, it used an API token intended for a limited function but which, in practice, granted broad permissions within the Railway infrastructure. Railway is a cloud platform that allows developers to deploy and manage applications without having to set up servers themselves.
A single API call deleted a storage volume. There was no additional verification, no confirmation step, and no separation between environments at the level of this action. Because backups were stored on the same volume, they were deleted simultaneously. The most recent restore point turned out to be months old.
The AI agent itself later indicated that it had made assumptions without verification, that it performed a destructive action without an explicit request, and that it lacked sufficient insight into the impact of the action. This reveals a structural problem: the security rules provided in prompts and configurations do not function as enforceable control mechanisms.
It is striking that this is not an exceptional setup. The tooling used is considered standard within development environments and is actively marketed for professional use. This shifts the discussion from the incident level to the system level.
AI Ignores Its Own Security Frameworks
The impact was immediately operational. PocketOS provides software to rental companies that rely on up-to-date data for reservations and customer management. Following the incident, recent bookings and customer data were missing, leading to disruptions that lasted for hours and required manual reconstruction. Data from the past few months could not be fully recovered.
The incident makes it clear that risks are not limited to the behavior of the AI agent itself. The underlying infrastructure plays an equally significant role. APIs that allow destructive actions without additional controls, tokens without a granular permission structure, and backups that are not isolated from production data significantly amplify the impact of errors or autonomous decisions.
For organizations that connect AI agents to production environments, this means that traditional assumptions about security are no longer sufficient. Model-level instructions offer no guarantee. Without enforceable restrictions at the API and infrastructure levels, a situation arises in which a single action, intended as a correction, can escalate into large-scale data loss.
