AI evaluations are meant to measure what a model can do. They can become real security incidents when the environment around the model is not actually isolated.

That is the lesson Simon Willison draws from Anthropic’s report on three incidents in its cybersecurity evaluations. Willison is an independent open source developer, the creator of Datasette, and a long-time writer about AI and software. He frames the pattern plainly:

“It happened again! This is turning into something of a pattern.”

The phrase is short, but the issue is serious. A model can be told that it is operating in a simulation. If the connected systems are real, the model may still treat them as available targets or tools. The prompt is not a security boundary.

What happened

Anthropic’s report describes three incidents found in a review of its cybersecurity evaluations. In each case, an evaluation prompt said the environment was a simulation and had no internet access. That assumption was wrong. The evaluation environment could reach real systems.

Willison also covered the earlier OpenAI incident in a technical timeline of the Hugging Face intrusion. The two posts point to the same failure pattern: a capable agent was given tools and network paths that were broader than the people running the test understood.

This is not mainly a story about a model “going rogue.” It is a story about a system that gave an agent access to something real, then relied on instructions to keep the agent inside an imaginary boundary.

Why Willison’s take carries weight

Willison has built open source software for data journalism and has written about web development and programming since 2002. His work often translates new AI capabilities into concrete examples that builders can inspect.

That makes his framing useful here. He is not arguing that every model is an attacker. He is showing why builders should judge the whole setup, including the model, tools, permissions, data, and network access.

What the analysis gets right

The clearest lesson is that a system prompt cannot replace isolation.

Isolation means putting a test agent in an environment where its actions cannot affect real accounts, services, or data. A sentence that says “this is only a simulation” does not create isolation. Neither does hiding a secret in a prompt or trusting the agent to recognize what is fictional.

The same distinction matters in ordinary AI products. A coding agent may have access to a repository, a deployment account, a package registry, a browser, or a database. Each connection expands what the agent can change. A builder needs to decide which connections are necessary before turning them on.

Willison’s analysis also helps separate model behavior from product responsibility. The model may make a bad decision, but the product owner decides whether that decision can reach a real system. A safe design assumes the model can misunderstand instructions and limits the damage when it does.

Where the framing is incomplete

The incident pattern does not mean every autonomous feature should be abandoned. It means “autonomous” needs a narrower definition than “the agent can do anything required to finish the task.”

Some tasks need access to real data or services. In those cases, builders can still reduce risk by limiting the agent’s permissions, separating test accounts from production accounts, requiring approval for irreversible actions, and recording what the agent did. These controls do not make a system harmless, but they make mistakes easier to contain and investigate.

There is also a trade-off. More approval steps can make an agent less useful. That is a product decision, not a purely technical one. The right question is whether the speed gained is worth the worst-case action the agent can take without review.

What builders should do

Before giving an agent a new tool or connection, ask:

  • Is the account or dataset separate from production?
  • Can the agent read, write, delete, or publish?
  • Which actions need a human approval?
  • What happens if the agent misunderstands the task?
  • Can we reconstruct its actions after the fact?

Start with the narrowest permission that supports the job. Use fake data and disposable accounts for evaluations. Keep deployment, billing, and customer data behind an explicit approval step. Treat external instructions, fetched pages, and files as untrusted input rather than as part of the agent’s trusted control plane.

Willison’s warning is not that models suddenly became uniquely malicious. It is that capability makes sloppy boundaries more expensive. If an agent can act, the environment must be designed as though its instructions can fail.

The practical rule is simple: test agents in worlds that are actually fake, and give production agents only the access their task can justify.

End of article