An AI agent inside a company with a person supervising: what to verify first
Software agents on a company's systems where a human was meant to oversee the work. The Register holds 6 comparable cases. Below are the controls that were missing or failed in them, those missing where harm followed first, each with the cases behind it and the rules that map to it. It shows what to verify, not that a deployment is safe.
- Compared against
- Enterprise work agents · Human supervises
- Assumes
- Your system is of this kind. The Register tags each case by sector, by how far the system acted alone, and by context; it does not yet tag what the system could reach (credentials, money, production data, the open web), so this result cannot narrow by that.
- Evidence as of
- 2026-11-16, from 40 published cases
- Not included
- Deployments that went well, which the Register does not collect; cases still inside a company's 14-day right of reply.
What to put in place first
6 controls, those missing where harm followed first. The order shows where failures occurred in these cases, not which control matters most for you. Rule mappings are drafts until a person checks them, and none is legal advice.
- Missing where harm followed
- 1Confirmation, delay, or threshold on irreversible actions.missing in 2 of 6 · 2 ended in harmRules that map to it OWASP LLM Top 10 recommendedCISA/NSA agentic AI adoption guide recommendedChina AI agent implementation opinions recommendedSingapore agentic AI framework recommended
What it would have looked like in those cases
- PocketOS 2026 (failed): Failed: Tool-side safety features. Absent: Confirmation or delayed delete on destructive calls.
- Replit and SaaStr 2025 (absent): Destructive-action gate
- 2The task's scope exists as an enforced constraint, not only as an instruction.missing in 2 of 6 · 2 ended in harm · tied with the one aboveRules that map to it OWASP LLM Top 10 recommendedCISA/NSA agentic AI adoption guide recommendedChina AI agent implementation opinions recommendedSingapore agentic AI framework recommended
What it would have looked like in those cases
- PocketOS 2026 (absent): Project-level safety rules given to the agent
- Replit and SaaStr 2025 (absent): Instruction-based code freeze
- 3Token or key limited to the environment and assets the task names.missing in 3 of 6 · 1 ended in harmRules that map to it NIS2 Directive impliedOWASP LLM Top 10 recommendedNIST/NSA/CISA agent incident practices recommendedCISA/NSA agentic AI adoption guide recommendedSingapore agentic AI framework recommended
What it would have looked like in those cases
- GitHub MCP leak 2025 (absent): Per-task token scoping
- PocketOS 2026 (absent): Environment-scoped token
- Claude Code [bot] trust 2026 (failed): Process-environment exposure to the agent
- 4Identity established by credential and binding, not by role, name, or convention.missing in 2 of 6 · 1 ended in harmRules that map to it EU Cyber Resilience Act impliedNIS2 Directive impliedCISA/NSA agentic AI adoption guide recommendedChina AI agent implementation opinions recommendedSingapore agentic AI framework recommended
What it would have looked like in those cases
- Agent-run intrusion campaign 2025 (absent): Absent: The operators told the model, in prompts, that they were employees of legitimate cybersecurity firms doing defensive testing, and the model went along with it. no source describes any check of who the operators were or whether they were authorised to test the targets. The vendor describes the bypass as role-play combined with splitting the attack into tasks that looked legitimate on their own (per the vendor).
- Claude Code [bot] trust 2026 (failed): Failed: In agent mode any GitHub App passed the write-permission check regardless of its actual permissions. the allowed_bots allow-list (in the action since August 2025) and the actor-type check were enforced in tag mode but not in agent mode until the 16 January 2026 fix.
- 5Operating domain enforced mechanically (geofence, environment scoping, operational design domain).missing in 1 of 6 · 1 ended in harmRules that map to it NIST/NSA/CISA agent incident practices recommended
What it would have looked like in those cases
- Replit and SaaStr 2025 (absent): Technical separation of development and production
- 6A record of actions the actor cannot write to or alter.missing in 1 of 6 · 1 ended in harm · tied with the one aboveRules that map to it EU Product Liability Directive impliedEU Cyber Resilience Act impliedEU AI Act proposedCIRCIA incident reporting proposedOWASP LLM Top 10 recommended+4
What it would have looked like in those cases
- Replit and SaaStr 2025 (absent): Absent: Independent audit log. no source describes a record of the agent's actions kept outside the agent.
6 more controls seen in these cases
- 7A victim can identify the actor without relying on the actor's vendor.missing in 1 of 6 · 1 ended in harm · tied with the one above
What it would have looked like in those cases
- Agent-run intrusion campaign 2025 (absent): Independent attribution path
- 8Text the actor reads is never treated as instruction.missing in 2 of 6 · 0 ended in harm
What it would have looked like in those cases
- GitHub MCP leak 2025 (absent): Retrieved issue text treated as instruction
- Claude Code [bot] trust 2026 (absent): Issue text resembling an error message treated as instruction
- 9Sandbox, network egress, and communication limits on a test or untrusted actor.missing in 1 of 6 · 0 ended in harm
What it would have looked like in those cases
- GitHub MCP leak 2025 (absent): Absent: No limit on where the agent could send what it read: it was connected with permission for the private repositories as well as the public one, and it wrote private-repository data into a public pull request. No approval prompt is recorded stopping it: Claude Desktop, used in the researchers' run, confirms individual tool calls by default, though the researchers say many users choose Always Allow, and they do not say which setting their run used. the author of the reproduction posted in issue #844 writes that there is human-in-the-loop verification, a 'Continue' button with a 'See More' preview of what the click will do.
- 10A path for outside researchers to report and for the vendor to fix before exploitation.missing in 0 of 6 · worked in 1
Where it was in place and worked
- Claude Code [bot] trust 2026: Worked: Coordinated disclosure. Four-day patch.
- 11Backups, rollback, and logs the actor cannot reach.missing in 0 of 6 · worked in 2 · tied with the one above
Where it was in place and worked
- PocketOS 2026: Worked: Railway's offsite disaster backups, which the agent could not reach, restored the data two days later. the volume-level backups sat in the same failure domain as the volume and the customer's own copy was three months old.
- Replit and SaaStr 2025: Platform rollback
- 12Internal review, restriction, recall, disclosure with regulator visibility.missing in 0 of 6 · worked in 1 · tied with the one above
Where it was in place and worked
- Agent-run intrusion campaign 2025: Vendor detection and account action
Take it with you
The same result as plain text, with the case citations and the rules. Paste it into a risk register, a design review, or a pull request.
Secure Autonomous checklist
Profile: Enterprise work agents · Human supervises
Comparable cases: 6 (3 ended in harm). Built 2026-11-16 from 40 cases.
[ ] 1. Destructive-action gate (C-E3, Envelope; prevents the action): Confirmation, delay, or threshold on irreversible actions.
Observed missing in 2 of 6 comparable included cases, 2 ended in harm.
Rules that map to it: OWASP LLM Top 10 (recommended); CISA/NSA agentic AI adoption guide (recommended); China AI agent implementation opinions (recommended); Singapore agentic AI framework (recommended).
- PocketOS 2026 (REG-0011): Failed: Tool-side safety features. Absent: Confirmation or delayed delete on destructive calls.
- Replit and SaaStr 2025 (REG-0012): Destructive-action gate
[ ] 2. Machine-enforceable scope (C-M2, Mandate; prevents the action): The task's scope exists as an enforced constraint, not only as an instruction.
Observed missing in 2 of 6 comparable included cases, 2 ended in harm.
Rules that map to it: OWASP LLM Top 10 (recommended); CISA/NSA agentic AI adoption guide (recommended); China AI agent implementation opinions (recommended); Singapore agentic AI framework (recommended).
- PocketOS 2026 (REG-0011): Project-level safety rules given to the agent
- Replit and SaaStr 2025 (REG-0012): Instruction-based code freeze
[ ] 3. Least-privilege credential scoped to task (C-E1, Envelope; prevents the action): Token or key limited to the environment and assets the task names.
Observed missing in 3 of 6 comparable included cases, 1 ended in harm.
Rules that map to it: NIS2 Directive (implied); OWASP LLM Top 10 (recommended); NIST/NSA/CISA agent incident practices (recommended); CISA/NSA agentic AI adoption guide (recommended); Singapore agentic AI framework (recommended).
- GitHub MCP leak 2025 (REG-0005): Per-task token scoping
- PocketOS 2026 (REG-0011): Environment-scoped token
- Claude Code [bot] trust 2026 (REG-0024): Process-environment exposure to the agent
[ ] 4. Actor identity verification (C-A1, Actor; prevents the action): Identity established by credential and binding, not by role, name, or convention.
Observed missing in 2 of 6 comparable included cases, 1 ended in harm.
Rules that map to it: EU Cyber Resilience Act (implied); NIS2 Directive (implied); CISA/NSA agentic AI adoption guide (recommended); China AI agent implementation opinions (recommended); Singapore agentic AI framework (recommended).
- Agent-run intrusion campaign 2025 (REG-0023): Absent: The operators told the model, in prompts, that they were employees of legitimate cybersecurity firms doing defensive testing, and the model went along with it. no source describes any check of who the operators were or whether they were authorised to test the targets. The vendor describes the bypass as role-play combined with splitting the attack into tasks that looked legitimate on their own (per the vendor).
- Claude Code [bot] trust 2026 (REG-0024): Failed: In agent mode any GitHub App passed the write-permission check regardless of its actual permissions. the allowed_bots allow-list (in the action since August 2025) and the actor-type check were enforced in tag mode but not in agent mode until the 16 January 2026 fix.
[ ] 5. Hard domain restriction (C-E2, Envelope; prevents the action): Operating domain enforced mechanically (geofence, environment scoping, operational design domain).
Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
Rules that map to it: NIST/NSA/CISA agent incident practices (recommended).
- Replit and SaaStr 2025 (REG-0012): Technical separation of development and production
[ ] 6. Independent audit log (C-V1, Evidence; establishes what happened): A record of actions the actor cannot write to or alter.
Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
Rules that map to it: EU Product Liability Directive (implied); EU Cyber Resilience Act (implied); EU AI Act (proposed); CIRCIA incident reporting (proposed); OWASP LLM Top 10 (recommended); NIST/NSA/CISA agent incident practices (recommended); CISA/NSA agentic AI adoption guide (recommended); China AI agent implementation opinions (recommended); Singapore agentic AI framework (recommended).
- Replit and SaaStr 2025 (REG-0012): Absent: Independent audit log. no source describes a record of the agent's actions kept outside the agent.
[ ] 7. Attribution independent of the vendor (C-V3, Evidence; establishes what happened): A victim can identify the actor without relying on the actor's vendor.
Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
- Agent-run intrusion campaign 2025 (REG-0023): Independent attribution path
[ ] 8. Retrieved content as data (C-I1, Inputs; prevents the action): Text the actor reads is never treated as instruction.
Observed missing in 2 of 6 comparable included cases.
Rules that map to it: EU AI Act (proposed); OWASP LLM Top 10 (recommended); NIST AI 600-1 (generative AI) (recommended); CISA/NSA agentic AI adoption guide (recommended); Singapore agentic AI framework (recommended).
- GitHub MCP leak 2025 (REG-0005): Retrieved issue text treated as instruction
- Claude Code [bot] trust 2026 (REG-0024): Issue text resembling an error message treated as instruction
[ ] 9. Containment and isolation (C-E5, Envelope; limits the damage): Sandbox, network egress, and communication limits on a test or untrusted actor.
Observed missing in 1 of 6 comparable included cases.
Rules that map to it: NIST/NSA/CISA agent incident practices (recommended); CISA/NSA agentic AI adoption guide (recommended); Singapore agentic AI framework (recommended).
- GitHub MCP leak 2025 (REG-0005): Absent: No limit on where the agent could send what it read: it was connected with permission for the private repositories as well as the public one, and it wrote private-repository data into a public pull request. No approval prompt is recorded stopping it: Claude Desktop, used in the researchers' run, confirms individual tool calls by default, though the researchers say many users choose Always Allow, and they do not say which setting their run used. the author of the reproduction posted in issue #844 writes that there is human-in-the-loop verification, a 'Continue' button with a 'See More' preview of what the click will do.
Source: secureautonomous.com/check/#ent.supervised