An AI agent doing work inside a company, in production: what to verify first
Software agents acting on a company's own systems, data or code, in live use. The Register holds 6 comparable cases. Below are the controls that were missing or failed in them, those missing where harm followed first, each with the cases behind it and the rules that map to it. It shows what to verify, not that a deployment is safe.
- Compared against
- Enterprise work agents · Production
- Assumes
- Your system is of this kind. The Register tags each case by sector, by how far the system acted alone, and by context; it does not yet tag what the system could reach (credentials, money, production data, the open web), so this result cannot narrow by that.
- Evidence as of
- 2026-11-16, from 40 published cases
- Not included
- Deployments that went well, which the Register does not collect; cases still inside a company's 14-day right of reply.
What to put in place first
6 controls, those missing where harm followed first. The order shows where failures occurred in these cases, not which control matters most for you. Rule mappings are drafts until a person checks them, and none is legal advice.
- Missing where harm followed
- 1Confirmation, delay, or threshold on irreversible actions.missing in 2 of 6 · 2 ended in harmRules that map to it OWASP LLM Top 10 recommendedCISA/NSA agentic AI adoption guide recommendedChina AI agent implementation opinions recommendedSingapore agentic AI framework recommended
What it would have looked like in those cases
- PocketOS 2026 (failed): Failed: Tool-side safety features. Absent: Confirmation or delayed delete on destructive calls.
- Replit and SaaStr 2025 (absent): Destructive-action gate
- 2The task's scope exists as an enforced constraint, not only as an instruction.missing in 2 of 6 · 2 ended in harm · tied with the one aboveRules that map to it OWASP LLM Top 10 recommendedCISA/NSA agentic AI adoption guide recommendedChina AI agent implementation opinions recommendedSingapore agentic AI framework recommended
What it would have looked like in those cases
- PocketOS 2026 (absent): Project-level safety rules given to the agent
- Replit and SaaStr 2025 (absent): Instruction-based code freeze
- 3Plan or command checked before it runs (review, twin, dry run).missing in 1 of 6 · 1 ended in harmRules that map to it OMB M-25-21 and M-25-22 required by the text
What it would have looked like in those cases
- Air Canada chatbot 2024 (absent): Absent: The record shows no check of the chatbot's policy answers against the airline's policy page before they reached the customer. the tribunal found Air Canada did not take reasonable care to ensure its chatbot was accurate, and Air Canada gave no information about the nature of its chatbot.
- 4The minimal-risk condition: when blocked or unsure, stop and ask.missing in 1 of 6 · 1 ended in harm · tied with the one aboveRules that map to it CISA/NSA agentic AI adoption guide recommended
What it would have looked like in those cases
- Anthropic's Claude Mythos 5, on a… (absent): Absent: Mechanism: the project required data the agency sold for a fee. the model, finding that the public dashboard issues an access token to any visitor, requested one and queried the database without paying (Anthropic's report). Why it interrupts: a rule that the agent stops and hands back to its researcher when data is offered only for a fee would have ended the task at the fee, before the token request. Source shows it absent: the report says of these cases 'Generally, this occurred when a server refused Claude's request, or when the data was offered only for a fee', and the spokesperson says the model acted 'Without being asked to'. no hand-off is described. The source does not show the counterfactual.
- 5Token or key limited to the environment and assets the task names.missing in 1 of 6 · 1 ended in harm · tied with the one aboveRules that map to it NIS2 Directive impliedOWASP LLM Top 10 recommendedNIST/NSA/CISA agent incident practices recommendedCISA/NSA agentic AI adoption guide recommendedSingapore agentic AI framework recommended
What it would have looked like in those cases
- PocketOS 2026 (absent): Environment-scoped token
- 6Operating domain enforced mechanically (geofence, environment scoping, operational design domain).missing in 1 of 6 · 1 ended in harm · tied with the one aboveRules that map to it NIST/NSA/CISA agent incident practices recommended
What it would have looked like in those cases
- Replit and SaaStr 2025 (absent): Technical separation of development and production
3 more controls seen in these cases
- 7A record of actions the actor cannot write to or alter.missing in 1 of 6 · 1 ended in harm · tied with the one above
What it would have looked like in those cases
- Replit and SaaStr 2025 (absent): Absent: Independent audit log. no source describes a record of the agent's actions kept outside the agent.
- 8Backups, rollback, and logs the actor cannot reach.missing in 0 of 6 · worked in 2
Where it was in place and worked
- PocketOS 2026: Worked: Railway's offsite disaster backups, which the agent could not reach, restored the data two days later. the volume-level backups sat in the same failure domain as the volume and the customer's own copy was three months old.
- Replit and SaaStr 2025: Platform rollback
- 9Internal review, restriction, recall, disclosure with regulator visibility.missing in 0 of 6 · worked in 1 · tied with the one above
Where it was in place and worked
- Meta's automated protections take the No…: Worked: Mechanism: an account was wrongly taken offline. Why it interrupts: an internal review that can reverse the action, plus public disclosure, limits how long the error lasts. Source shows it succeeded in part: Meta says an automated re-review recognized the error an hour later and restored the account (the coalition says the page was hidden for about 90 minutes). Meta apologized on 17 October and gave a revised explanation on 18 October. Meta says restoring administrators' access 'took additional time', which the coalition puts at about four hours in all.
Take it with you
The same result as plain text, with the case citations and the rules. Paste it into a risk register, a design review, or a pull request.
Secure Autonomous checklist
Profile: Enterprise work agents · Production
Comparable cases: 6 (5 ended in harm). Built 2026-11-16 from 40 cases.
[ ] 1. Destructive-action gate (C-E3, Envelope; prevents the action): Confirmation, delay, or threshold on irreversible actions.
Observed missing in 2 of 6 comparable included cases, 2 ended in harm.
Rules that map to it: OWASP LLM Top 10 (recommended); CISA/NSA agentic AI adoption guide (recommended); China AI agent implementation opinions (recommended); Singapore agentic AI framework (recommended).
- PocketOS 2026 (REG-0011): Failed: Tool-side safety features. Absent: Confirmation or delayed delete on destructive calls.
- Replit and SaaStr 2025 (REG-0012): Destructive-action gate
[ ] 2. Machine-enforceable scope (C-M2, Mandate; prevents the action): The task's scope exists as an enforced constraint, not only as an instruction.
Observed missing in 2 of 6 comparable included cases, 2 ended in harm.
Rules that map to it: OWASP LLM Top 10 (recommended); CISA/NSA agentic AI adoption guide (recommended); China AI agent implementation opinions (recommended); Singapore agentic AI framework (recommended).
- PocketOS 2026 (REG-0011): Project-level safety rules given to the agent
- Replit and SaaStr 2025 (REG-0012): Instruction-based code freeze
[ ] 3. Pre-execution validation (C-D1, Decision; prevents the action): Plan or command checked before it runs (review, twin, dry run).
Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
Rules that map to it: OMB M-25-21 and M-25-22 (required by the text).
- Air Canada chatbot 2024 (REG-0010): Absent: The record shows no check of the chatbot's policy answers against the airline's policy page before they reached the customer. the tribunal found Air Canada did not take reasonable care to ensure its chatbot was accurate, and Air Canada gave no information about the nature of its chatbot.
[ ] 4. Halt and hand off on uncertainty or refusal (C-D2, Decision; prevents the action): The minimal-risk condition: when blocked or unsure, stop and ask.
Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
Rules that map to it: CISA/NSA agentic AI adoption guide (recommended).
- Anthropic's Claude Mythos 5, on a… (REG-0042): Absent: Mechanism: the project required data the agency sold for a fee. the model, finding that the public dashboard issues an access token to any visitor, requested one and queried the database without paying (Anthropic's report). Why it interrupts: a rule that the agent stops and hands back to its researcher when data is offered only for a fee would have ended the task at the fee, before the token request. Source shows it absent: the report says of these cases 'Generally, this occurred when a server refused Claude's request, or when the data was offered only for a fee', and the spokesperson says the model acted 'Without being asked to'. no hand-off is described. The source does not show the counterfactual.
[ ] 5. Least-privilege credential scoped to task (C-E1, Envelope; prevents the action): Token or key limited to the environment and assets the task names.
Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
Rules that map to it: NIS2 Directive (implied); OWASP LLM Top 10 (recommended); NIST/NSA/CISA agent incident practices (recommended); CISA/NSA agentic AI adoption guide (recommended); Singapore agentic AI framework (recommended).
- PocketOS 2026 (REG-0011): Environment-scoped token
[ ] 6. Hard domain restriction (C-E2, Envelope; prevents the action): Operating domain enforced mechanically (geofence, environment scoping, operational design domain).
Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
Rules that map to it: NIST/NSA/CISA agent incident practices (recommended).
- Replit and SaaStr 2025 (REG-0012): Technical separation of development and production
[ ] 7. Independent audit log (C-V1, Evidence; establishes what happened): A record of actions the actor cannot write to or alter.
Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
Rules that map to it: EU Product Liability Directive (implied); EU Cyber Resilience Act (implied); EU AI Act (proposed); CIRCIA incident reporting (proposed); OWASP LLM Top 10 (recommended); NIST/NSA/CISA agent incident practices (recommended); CISA/NSA agentic AI adoption guide (recommended); China AI agent implementation opinions (recommended); Singapore agentic AI framework (recommended).
- Replit and SaaStr 2025 (REG-0012): Absent: Independent audit log. no source describes a record of the agent's actions kept outside the agent.
Source: secureautonomous.com/check/#ent.production