Secure Autonomous
FeedbackGet alertsCheck a deployment
Check › Starting point

An AI agent inside a company with a person supervising: what to verify first

Software agents on a company's systems where a human was meant to oversee the work. The Register holds 6 comparable cases. Below are the controls that were missing or failed in them, those missing where harm followed first, each with the cases behind it and the rules that map to it. It shows what to verify, not that a deployment is safe.

Compared against
Enterprise work agents · Human supervises
Assumes
Your system is of this kind. The Register tags each case by sector, by how far the system acted alone, and by context; it does not yet tag what the system could reach (credentials, money, production data, the open web), so this result cannot narrow by that.
Evidence as of
2026-11-16, from 40 published cases
Not included
Deployments that went well, which the Register does not collect; cases still inside a company's 14-day right of reply.
6comparable cases in the Register: 3 ended in harmEnterprise work agents · Human supervises
3 ended in harm2 caught first by researchers1 pending

What to put in place first

6 controls, those missing where harm followed first. The order shows where failures occurred in these cases, not which control matters most for you. Rule mappings are drafts until a person checks them, and none is legal advice.

  1. Missing where harm followed
  2. 1
    Destructive-action gate Prevents the action Envelope
    Confirmation, delay, or threshold on irreversible actions.
    missing in 2 of 6 · 2 ended in harm
    What it would have looked like in those cases
    • PocketOS 2026 (failed): Failed: Tool-side safety features. Absent: Confirmation or delayed delete on destructive calls.
    • Replit and SaaStr 2025 (absent): Destructive-action gate
  3. 2
    Machine-enforceable scope Prevents the action Mandate
    The task's scope exists as an enforced constraint, not only as an instruction.
    missing in 2 of 6 · 2 ended in harm · tied with the one above
    What it would have looked like in those cases
  4. 3
    Least-privilege credential scoped to task Prevents the action Envelope
    Token or key limited to the environment and assets the task names.
    missing in 3 of 6 · 1 ended in harm
    What it would have looked like in those cases
  5. 4
    Actor identity verification Prevents the action Actor
    Identity established by credential and binding, not by role, name, or convention.
    missing in 2 of 6 · 1 ended in harm
    What it would have looked like in those cases
    • Agent-run intrusion campaign 2025 (absent): Absent: The operators told the model, in prompts, that they were employees of legitimate cybersecurity firms doing defensive testing, and the model went along with it. no source describes any check of who the operators were or whether they were authorised to test the targets. The vendor describes the bypass as role-play combined with splitting the attack into tasks that looked legitimate on their own (per the vendor).
    • Claude Code [bot] trust 2026 (failed): Failed: In agent mode any GitHub App passed the write-permission check regardless of its actual permissions. the allowed_bots allow-list (in the action since August 2025) and the actor-type check were enforced in tag mode but not in agent mode until the 16 January 2026 fix.
  6. 5
    Hard domain restriction Prevents the action Envelope
    Operating domain enforced mechanically (geofence, environment scoping, operational design domain).
    missing in 1 of 6 · 1 ended in harm
    What it would have looked like in those cases
  7. 6
    Independent audit log Establishes what happened Evidence
    A record of actions the actor cannot write to or alter.
    missing in 1 of 6 · 1 ended in harm · tied with the one above
    What it would have looked like in those cases
    • Replit and SaaStr 2025 (absent): Absent: Independent audit log. no source describes a record of the agent's actions kept outside the agent.

6 more controls seen in these cases

  1. 7
    Attribution independent of the vendor Establishes what happened Evidence
    A victim can identify the actor without relying on the actor's vendor.
    missing in 1 of 6 · 1 ended in harm · tied with the one above
    What it would have looked like in those cases
  2. 8
    Retrieved content as data Prevents the action Inputs
    Text the actor reads is never treated as instruction.
    missing in 2 of 6 · 0 ended in harm
    What it would have looked like in those cases
  3. 9
    Containment and isolation Limits the damage Envelope
    Sandbox, network egress, and communication limits on a test or untrusted actor.
    missing in 1 of 6 · 0 ended in harm
    What it would have looked like in those cases
    • GitHub MCP leak 2025 (absent): Absent: No limit on where the agent could send what it read: it was connected with permission for the private repositories as well as the public one, and it wrote private-repository data into a public pull request. No approval prompt is recorded stopping it: Claude Desktop, used in the researchers' run, confirms individual tool calls by default, though the researchers say many users choose Always Allow, and they do not say which setting their run used. the author of the reproduction posted in issue #844 writes that there is human-in-the-loop verification, a 'Continue' button with a 'See More' preview of what the click will do.
  4. 10
    Coordinated disclosure program Detects it External
    A path for outside researchers to report and for the vendor to fix before exploitation.
    missing in 0 of 6 · worked in 1
    Where it was in place and worked
  5. 11
    Backups, rollback, and logs the actor cannot reach.
    missing in 0 of 6 · worked in 2 · tied with the one above
    Where it was in place and worked
    • PocketOS 2026: Worked: Railway's offsite disaster backups, which the agent could not reach, restored the data two days later. the volume-level backups sat in the same failure domain as the volume and the customer's own copy was three months old.
    • Replit and SaaStr 2025: Platform rollback
  6. 12
    Internal review, restriction, recall, disclosure with regulator visibility.
    missing in 0 of 6 · worked in 1 · tied with the one above
    Where it was in place and worked
How to read this list. A count shows how often a control was missing or failed in the 6 comparable cases the Register holds. It does not show how well the control works, how likely a failure is, or how risky your deployment is: the Register is a selected set of documented cases, with no record of deployments that went well. This is not a risk score. Controls are ordered by the number of comparable cases that ended in harm, then by the number of comparable cases in all; where two controls have the same counts they are marked as tied, and the order between them carries no meaning. Whether a rule applies to you depends on facts this page does not know; the mappings show which control a rule's text calls for, not that the rule binds your organization.

Take it with you

The same result as plain text, with the case citations and the rules. Paste it into a risk register, a design review, or a pull request.

Change the selection Alert me when this changes

Secure Autonomous checklist
Profile: Enterprise work agents · Human supervises
Comparable cases: 6 (3 ended in harm). Built 2026-11-16 from 40 cases.

[ ] 1. Destructive-action gate (C-E3, Envelope; prevents the action): Confirmation, delay, or threshold on irreversible actions.
    Observed missing in 2 of 6 comparable included cases, 2 ended in harm.
    Rules that map to it: OWASP LLM Top 10 (recommended); CISA/NSA agentic AI adoption guide (recommended); China AI agent implementation opinions (recommended); Singapore agentic AI framework (recommended).
    - PocketOS 2026 (REG-0011): Failed: Tool-side safety features. Absent: Confirmation or delayed delete on destructive calls.
    - Replit and SaaStr 2025 (REG-0012): Destructive-action gate

[ ] 2. Machine-enforceable scope (C-M2, Mandate; prevents the action): The task's scope exists as an enforced constraint, not only as an instruction.
    Observed missing in 2 of 6 comparable included cases, 2 ended in harm.
    Rules that map to it: OWASP LLM Top 10 (recommended); CISA/NSA agentic AI adoption guide (recommended); China AI agent implementation opinions (recommended); Singapore agentic AI framework (recommended).
    - PocketOS 2026 (REG-0011): Project-level safety rules given to the agent
    - Replit and SaaStr 2025 (REG-0012): Instruction-based code freeze

[ ] 3. Least-privilege credential scoped to task (C-E1, Envelope; prevents the action): Token or key limited to the environment and assets the task names.
    Observed missing in 3 of 6 comparable included cases, 1 ended in harm.
    Rules that map to it: NIS2 Directive (implied); OWASP LLM Top 10 (recommended); NIST/NSA/CISA agent incident practices (recommended); CISA/NSA agentic AI adoption guide (recommended); Singapore agentic AI framework (recommended).
    - GitHub MCP leak 2025 (REG-0005): Per-task token scoping
    - PocketOS 2026 (REG-0011): Environment-scoped token
    - Claude Code [bot] trust 2026 (REG-0024): Process-environment exposure to the agent

[ ] 4. Actor identity verification (C-A1, Actor; prevents the action): Identity established by credential and binding, not by role, name, or convention.
    Observed missing in 2 of 6 comparable included cases, 1 ended in harm.
    Rules that map to it: EU Cyber Resilience Act (implied); NIS2 Directive (implied); CISA/NSA agentic AI adoption guide (recommended); China AI agent implementation opinions (recommended); Singapore agentic AI framework (recommended).
    - Agent-run intrusion campaign 2025 (REG-0023): Absent: The operators told the model, in prompts, that they were employees of legitimate cybersecurity firms doing defensive testing, and the model went along with it. no source describes any check of who the operators were or whether they were authorised to test the targets. The vendor describes the bypass as role-play combined with splitting the attack into tasks that looked legitimate on their own (per the vendor).
    - Claude Code [bot] trust 2026 (REG-0024): Failed: In agent mode any GitHub App passed the write-permission check regardless of its actual permissions. the allowed_bots allow-list (in the action since August 2025) and the actor-type check were enforced in tag mode but not in agent mode until the 16 January 2026 fix.

[ ] 5. Hard domain restriction (C-E2, Envelope; prevents the action): Operating domain enforced mechanically (geofence, environment scoping, operational design domain).
    Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
    Rules that map to it: NIST/NSA/CISA agent incident practices (recommended).
    - Replit and SaaStr 2025 (REG-0012): Technical separation of development and production

[ ] 6. Independent audit log (C-V1, Evidence; establishes what happened): A record of actions the actor cannot write to or alter.
    Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
    Rules that map to it: EU Product Liability Directive (implied); EU Cyber Resilience Act (implied); EU AI Act (proposed); CIRCIA incident reporting (proposed); OWASP LLM Top 10 (recommended); NIST/NSA/CISA agent incident practices (recommended); CISA/NSA agentic AI adoption guide (recommended); China AI agent implementation opinions (recommended); Singapore agentic AI framework (recommended).
    - Replit and SaaStr 2025 (REG-0012): Absent: Independent audit log. no source describes a record of the agent's actions kept outside the agent.

[ ] 7. Attribution independent of the vendor (C-V3, Evidence; establishes what happened): A victim can identify the actor without relying on the actor's vendor.
    Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
    - Agent-run intrusion campaign 2025 (REG-0023): Independent attribution path

[ ] 8. Retrieved content as data (C-I1, Inputs; prevents the action): Text the actor reads is never treated as instruction.
    Observed missing in 2 of 6 comparable included cases.
    Rules that map to it: EU AI Act (proposed); OWASP LLM Top 10 (recommended); NIST AI 600-1 (generative AI) (recommended); CISA/NSA agentic AI adoption guide (recommended); Singapore agentic AI framework (recommended).
    - GitHub MCP leak 2025 (REG-0005): Retrieved issue text treated as instruction
    - Claude Code [bot] trust 2026 (REG-0024): Issue text resembling an error message treated as instruction

[ ] 9. Containment and isolation (C-E5, Envelope; limits the damage): Sandbox, network egress, and communication limits on a test or untrusted actor.
    Observed missing in 1 of 6 comparable included cases.
    Rules that map to it: NIST/NSA/CISA agent incident practices (recommended); CISA/NSA agentic AI adoption guide (recommended); Singapore agentic AI framework (recommended).
    - GitHub MCP leak 2025 (REG-0005): Absent: No limit on where the agent could send what it read: it was connected with permission for the private repositories as well as the public one, and it wrote private-repository data into a public pull request. No approval prompt is recorded stopping it: Claude Desktop, used in the researchers' run, confirms individual tool calls by default, though the researchers say many users choose Always Allow, and they do not say which setting their run used. the author of the reproduction posted in issue #844 writes that there is human-in-the-loop verification, a 'Continue' button with a 'See More' preview of what the click will do.

Source: secureautonomous.com/check/#ent.supervised

Help shape the Register

A person reads every answer; changes it leads to are logged in the methodology. To dispute a claim or reply for a named company, use Respond, dispute, or submit.

Did this page help with what you came to do?