More AI Models Are Hacking Outside Companies, But AI Developers Keep Leaving Gates Open

Following on from the recent story of an OpenAI agent independently hacking Hugging Face and customer accounts at several other online services, Anthropic has reviewed Claude activity during security testing over the past four months and found it has also had at least three instances of escaping the backyard and roaming the internet committing breaches on its own.

But talk of an “AI apocalypse” may be premature. In all three cases the AI models were errantly given internet access when they were not supposed to have it, and were following instructions that did not specifically forbid them from taking certain aggressive actions.

Anthropic AI models reported as rogue after OpenAI dominates headlines

AI models jumping out of sandboxes and hacking third parties is a matter that is very easily sensationalized, so there are some important points of note about the incidents to address: the models all believed that they were in a simulation and pursuing their assigned end goals during at least the initial phases of the breaches, they were left a path to internet access by mistake (rather than hacking their way to it), and two of them stopped attacks after concluding their assumed simulated internet access was actually real.

The concern is thus less about Skynet developing and more about the handling of these tests by AI developers and their third-party partners (with a partner in this case leaving the gate open to the internet in all three incidents). Anthropic has contracted with METR for a full third-party review of the incident, but notes in its initial public report that some simple fixes (such as being more clear in its prompts to the AI models about how to handle unexpected internet access, or more thorough review of potential paths to such access) would have headed off the attacks.

None of the AI models, including the one that breached Hugging Face, was really operating with a “mind of its own.” They were all rigidly attempting to solve a security puzzle that their human handlers set them to, and simply did not recognize appropriate limits in that pursuit. Thus, aside from the speed and capability with which they were able to uncover vulnerabilities and capitalize on them, the main concern that is developing is whether or not their developers are adequately securing them.

Can AI developers independently restrain their models?

One other interesting point about the Anthropic attacks is that while all involved Claude, it was backed by three different models in each of the three different cases. Each of these models acted a little bit differently upon realizing that they were no longer in a simulation and were in fact on the open internet breaching a real target.

The oldest and ostensibly least capable of the AI models, Opus 4.7, was the most reckless and relentless along with proving that it was capable of finding and exploiting vulnerabilities and harvesting target data independently. After it wrongly assumed that a company it found on the internet with a similar name to its target was part of the simulation, it successfully stole access credentials and exfiltrated a production database. This was the model that also did not independently stop its attack: it did eventually reason that it was on the open internet and attacking a real company, but then further reasoned the company must be part of the simulation and an intended target.

A Mythos 5 Claude also engaged in an attack, but only under the assumption that its internet access was simulated. It breached a security company by uploading a tainted Python package designed to act as a trojan horse when examined for malware. However, its internal notes indicate it recognized that it should not do this on the real internet.

The last of the AI models is an unnamed project still in testing. This one wound up in a situation similar to that of Opus 4.7, scanning the external internet for lookalike companies and eventually finding one it assumed was a valid target. However, after initially breaching it, the model independently recognized it was committing a live attack on an actual organization and self-terminated its activity.

Anthropic believes some relatively simple improvements to monitoring and model alignment will prevent these rampages in the future. The cybersecurity community is not as convinced. The slow response and seeming lack of awareness has prompted calls for both greater access for security experts as well as regulation updated to assess responsibility when AI models go rogue. The spark will likely be the first incident in which a model’s self-directed hacking causes real world damage or financial harm.