A recent Tom’s Hardware article shared a striking story: an AI coding agent reportedly deleted a company’s production database and volume-level backups in a single API call. The company, PocketOS, said the deletion took only nine seconds and affected months of business-critical customer data. The article explains that the issue was not only the AI tool itself, but also the surrounding system design: destructive API access, backups connected to the same vulnerable environment, and limited recovery options.
At first glance, this sounds like a technology story.
But it is also a health and safety story.
Because the core issue is not “AI went rogue.” The real issue is something safety professionals understand very well: a high-risk task was completed without adequate controls, verification, supervision, or recovery planning.
AI Is Not the Hazard. Uncontrolled AI Is.
In health and safety, we rarely say a tool is the problem by itself. A forklift is not automatically dangerous. A grinder is not automatically dangerous. A confined space entry is not automatically a disaster.
The problem starts when the tool is used without a proper system around it.
The same applies to AI.
AI tools can improve speed, reduce administrative burden, support documentation, summarize information, create templates, and help businesses work more efficiently. For small and medium-sized businesses, that can be a major advantage. But when AI is given access to critical systems without boundaries, it becomes an operational risk.
In this case, the AI agent was reportedly working on a routine staging task but took destructive action that affected production data. The founder later described several failures, including the agent guessing instead of verifying and acting without asking for approval before taking a destructive step.
The Hierarchy of Controls Still Applies
When businesses introduce AI, they should treat it like any other new process, technology, or equipment. That means asking a basic safety question:
What could go wrong, and what controls are in place before we use it?
Using a safety lens, AI controls may look like this:
Elimination: Do not allow AI to access production systems or sensitive data where it is not required.
Substitution: Use AI for low-risk administrative tasks before allowing it into higher-risk workflows.
Engineering Controls: Build technical barriers, such as read-only access, sandbox environments, permission limits, approval gates, and system separation between development, staging, and production.
Administrative Controls: Create an AI use policy, define approved tasks, train users, require human review, and document when AI can and cannot be used.
PPE Equivalent: In digital work, this may look like backups, version history, recovery procedures, audit logs, and emergency response plans. It is the last layer, not the only layer.
The Tom’s Hardware article noted that the company did have an older backup, but the gap still left customers and the business reconstructing information manually from payment histories, calendar integrations, and email confirmations. That is the same problem we see in workplace safety when emergency plans exist on paper, but the practical recovery process has not been tested.
“Trust But Verify” Is Not Enough
A common phrase in safety is “trust but verify.”
With AI, that is not strong enough.
A better phrase is:
Design the system assuming the tool may make a confident mistake.
AI does not always know when it is wrong. It may produce a polished answer, make a recommendation, summarize a document, or complete an action with confidence. That does not mean the output is accurate, safe, compliant, or appropriate for the business.
For employers, this matters because AI is no longer limited to tech companies. It is being used for policy writing, safety manuals, training materials, claims management, documentation, client communication, hazard assessments, and administrative workflows.
That does not mean organizations should avoid AI. It means they need a management system around it.
Practical Questions Every Business Should Ask
Before using AI in safety, operations, or administration, businesses should ask:
What is AI allowed to do?
There should be clear boundaries between drafting, recommending, editing, and taking action.
What is AI not allowed to do?
Some tasks should remain human-controlled, especially anything involving legal decisions, discipline, medical information, claims decisions, production systems, or confidential data.
Who reviews the output?
AI can support safety documentation, but a competent person still needs to review it for accuracy, site-specific application, and legal alignment.
What information should never be entered?
Businesses should be careful with personal information, medical information, client data, proprietary information, and sensitive incident details.
What happens if the tool makes a mistake?
There should be a recovery plan. This includes backups, file versioning, access controls, and a clear process to correct errors quickly.
Has the team been trained?
Workers and supervisors need simple guidance. They do not need a 40-page AI policy. They need clear rules they can actually follow.
The Safety Professional’s Role Is Changing
The role of the safety professional has always been broader than paperwork. Safety professionals help organizations understand risk before it becomes an incident.
AI is simply another example of that.
The businesses that succeed with AI will not be the ones that ban it entirely or allow it without limits. They will be the ones that build practical guardrails around it.
That is where health and safety thinking becomes valuable.
Safety professionals are trained to look at systems. We ask whether workers are competent, whether controls are effective, whether emergency plans are realistic, whether supervision exists, and whether the organization can prove due diligence.
Those same questions apply to AI.
The Primera Perspective
At Primera Consulting, we believe safety should fit the business. That includes helping organizations use new tools responsibly, without creating unnecessary fear or unnecessary complexity.
AI can be an excellent support tool. It can help reduce administrative load, improve communication, and make safety systems easier to manage. But like any tool, it needs limits, review, and practical controls.
The lesson from this database deletion is not that AI should be avoided.
The lesson is that speed without control creates risk.
In health and safety, we already know this. A task completed quickly is not successful if it creates greater harm, greater exposure, or a bigger recovery problem later.
AI should make work easier. It should not remove accountability, verification, or common sense from the process.
