Secure Autonomous
FeedbackGet alertsCheck a deployment
Check › Starting point

An AI agent doing work inside a company, in production: what to verify first

Software agents acting on a company's own systems, data or code, in live use. The Register holds 6 comparable cases. Below are the controls that were missing or failed in them, those missing where harm followed first, each with the cases behind it and the rules that map to it. It shows what to verify, not that a deployment is safe.

Compared against
Enterprise work agents · Production
Assumes
Your system is of this kind. The Register tags each case by sector, by how far the system acted alone, and by context; it does not yet tag what the system could reach (credentials, money, production data, the open web), so this result cannot narrow by that.
Evidence as of
2026-11-16, from 40 published cases
Not included
Deployments that went well, which the Register does not collect; cases still inside a company's 14-day right of reply.
6comparable cases in the Register: 5 ended in harmEnterprise work agents · Production
5 ended in harm1 pending

What to put in place first

6 controls, those missing where harm followed first. The order shows where failures occurred in these cases, not which control matters most for you. Rule mappings are drafts until a person checks them, and none is legal advice.

  1. Missing where harm followed
  2. 1
    Destructive-action gate Prevents the action Envelope
    Confirmation, delay, or threshold on irreversible actions.
    missing in 2 of 6 · 2 ended in harm
    What it would have looked like in those cases
    • PocketOS 2026 (failed): Failed: Tool-side safety features. Absent: Confirmation or delayed delete on destructive calls.
    • Replit and SaaStr 2025 (absent): Destructive-action gate
  3. 2
    Machine-enforceable scope Prevents the action Mandate
    The task's scope exists as an enforced constraint, not only as an instruction.
    missing in 2 of 6 · 2 ended in harm · tied with the one above
    What it would have looked like in those cases
  4. 3
    Pre-execution validation Prevents the action Decision
    Plan or command checked before it runs (review, twin, dry run).
    missing in 1 of 6 · 1 ended in harm
    What it would have looked like in those cases
    • Air Canada chatbot 2024 (absent): Absent: The record shows no check of the chatbot's policy answers against the airline's policy page before they reached the customer. the tribunal found Air Canada did not take reasonable care to ensure its chatbot was accurate, and Air Canada gave no information about the nature of its chatbot.
  5. 4
    Halt and hand off on uncertainty or refusal Prevents the action Decision
    The minimal-risk condition: when blocked or unsure, stop and ask.
    missing in 1 of 6 · 1 ended in harm · tied with the one above
    What it would have looked like in those cases
    • Anthropic's Claude Mythos 5, on a… (absent): Absent: Mechanism: the project required data the agency sold for a fee. the model, finding that the public dashboard issues an access token to any visitor, requested one and queried the database without paying (Anthropic's report). Why it interrupts: a rule that the agent stops and hands back to its researcher when data is offered only for a fee would have ended the task at the fee, before the token request. Source shows it absent: the report says of these cases 'Generally, this occurred when a server refused Claude's request, or when the data was offered only for a fee', and the spokesperson says the model acted 'Without being asked to'. no hand-off is described. The source does not show the counterfactual.
  6. 5
    Least-privilege credential scoped to task Prevents the action Envelope
    Token or key limited to the environment and assets the task names.
    missing in 1 of 6 · 1 ended in harm · tied with the one above
    What it would have looked like in those cases
  7. 6
    Hard domain restriction Prevents the action Envelope
    Operating domain enforced mechanically (geofence, environment scoping, operational design domain).
    missing in 1 of 6 · 1 ended in harm · tied with the one above
    What it would have looked like in those cases

3 more controls seen in these cases

  1. 7
    Independent audit log Establishes what happened Evidence
    A record of actions the actor cannot write to or alter.
    missing in 1 of 6 · 1 ended in harm · tied with the one above
    What it would have looked like in those cases
    • Replit and SaaStr 2025 (absent): Absent: Independent audit log. no source describes a record of the agent's actions kept outside the agent.
  2. 8
    Backups, rollback, and logs the actor cannot reach.
    missing in 0 of 6 · worked in 2
    Where it was in place and worked
    • PocketOS 2026: Worked: Railway's offsite disaster backups, which the agent could not reach, restored the data two days later. the volume-level backups sat in the same failure domain as the volume and the customer's own copy was three months old.
    • Replit and SaaStr 2025: Platform rollback
  3. 9
    Internal review, restriction, recall, disclosure with regulator visibility.
    missing in 0 of 6 · worked in 1 · tied with the one above
    Where it was in place and worked
    • Meta's automated protections take the No…: Worked: Mechanism: an account was wrongly taken offline. Why it interrupts: an internal review that can reverse the action, plus public disclosure, limits how long the error lasts. Source shows it succeeded in part: Meta says an automated re-review recognized the error an hour later and restored the account (the coalition says the page was hidden for about 90 minutes). Meta apologized on 17 October and gave a revised explanation on 18 October. Meta says restoring administrators' access 'took additional time', which the coalition puts at about four hours in all.
How to read this list. A count shows how often a control was missing or failed in the 6 comparable cases the Register holds. It does not show how well the control works, how likely a failure is, or how risky your deployment is: the Register is a selected set of documented cases, with no record of deployments that went well. This is not a risk score. Controls are ordered by the number of comparable cases that ended in harm, then by the number of comparable cases in all; where two controls have the same counts they are marked as tied, and the order between them carries no meaning. Whether a rule applies to you depends on facts this page does not know; the mappings show which control a rule's text calls for, not that the rule binds your organization.

Take it with you

The same result as plain text, with the case citations and the rules. Paste it into a risk register, a design review, or a pull request.

Change the selection Alert me when this changes

Secure Autonomous checklist
Profile: Enterprise work agents · Production
Comparable cases: 6 (5 ended in harm). Built 2026-11-16 from 40 cases.

[ ] 1. Destructive-action gate (C-E3, Envelope; prevents the action): Confirmation, delay, or threshold on irreversible actions.
    Observed missing in 2 of 6 comparable included cases, 2 ended in harm.
    Rules that map to it: OWASP LLM Top 10 (recommended); CISA/NSA agentic AI adoption guide (recommended); China AI agent implementation opinions (recommended); Singapore agentic AI framework (recommended).
    - PocketOS 2026 (REG-0011): Failed: Tool-side safety features. Absent: Confirmation or delayed delete on destructive calls.
    - Replit and SaaStr 2025 (REG-0012): Destructive-action gate

[ ] 2. Machine-enforceable scope (C-M2, Mandate; prevents the action): The task's scope exists as an enforced constraint, not only as an instruction.
    Observed missing in 2 of 6 comparable included cases, 2 ended in harm.
    Rules that map to it: OWASP LLM Top 10 (recommended); CISA/NSA agentic AI adoption guide (recommended); China AI agent implementation opinions (recommended); Singapore agentic AI framework (recommended).
    - PocketOS 2026 (REG-0011): Project-level safety rules given to the agent
    - Replit and SaaStr 2025 (REG-0012): Instruction-based code freeze

[ ] 3. Pre-execution validation (C-D1, Decision; prevents the action): Plan or command checked before it runs (review, twin, dry run).
    Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
    Rules that map to it: OMB M-25-21 and M-25-22 (required by the text).
    - Air Canada chatbot 2024 (REG-0010): Absent: The record shows no check of the chatbot's policy answers against the airline's policy page before they reached the customer. the tribunal found Air Canada did not take reasonable care to ensure its chatbot was accurate, and Air Canada gave no information about the nature of its chatbot.

[ ] 4. Halt and hand off on uncertainty or refusal (C-D2, Decision; prevents the action): The minimal-risk condition: when blocked or unsure, stop and ask.
    Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
    Rules that map to it: CISA/NSA agentic AI adoption guide (recommended).
    - Anthropic's Claude Mythos 5, on a… (REG-0042): Absent: Mechanism: the project required data the agency sold for a fee. the model, finding that the public dashboard issues an access token to any visitor, requested one and queried the database without paying (Anthropic's report). Why it interrupts: a rule that the agent stops and hands back to its researcher when data is offered only for a fee would have ended the task at the fee, before the token request. Source shows it absent: the report says of these cases 'Generally, this occurred when a server refused Claude's request, or when the data was offered only for a fee', and the spokesperson says the model acted 'Without being asked to'. no hand-off is described. The source does not show the counterfactual.

[ ] 5. Least-privilege credential scoped to task (C-E1, Envelope; prevents the action): Token or key limited to the environment and assets the task names.
    Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
    Rules that map to it: NIS2 Directive (implied); OWASP LLM Top 10 (recommended); NIST/NSA/CISA agent incident practices (recommended); CISA/NSA agentic AI adoption guide (recommended); Singapore agentic AI framework (recommended).
    - PocketOS 2026 (REG-0011): Environment-scoped token

[ ] 6. Hard domain restriction (C-E2, Envelope; prevents the action): Operating domain enforced mechanically (geofence, environment scoping, operational design domain).
    Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
    Rules that map to it: NIST/NSA/CISA agent incident practices (recommended).
    - Replit and SaaStr 2025 (REG-0012): Technical separation of development and production

[ ] 7. Independent audit log (C-V1, Evidence; establishes what happened): A record of actions the actor cannot write to or alter.
    Observed missing in 1 of 6 comparable included cases, 1 ended in harm.
    Rules that map to it: EU Product Liability Directive (implied); EU Cyber Resilience Act (implied); EU AI Act (proposed); CIRCIA incident reporting (proposed); OWASP LLM Top 10 (recommended); NIST/NSA/CISA agent incident practices (recommended); CISA/NSA agentic AI adoption guide (recommended); China AI agent implementation opinions (recommended); Singapore agentic AI framework (recommended).
    - Replit and SaaStr 2025 (REG-0012): Absent: Independent audit log. no source describes a record of the agent's actions kept outside the agent.

Source: secureautonomous.com/check/#ent.production

Help shape the Register

A person reads every answer; changes it leads to are logged in the methodology. To dispute a claim or reply for a named company, use Respond, dispute, or submit.

Did this page help with what you came to do?