New Security Safeguards for OpenAI May Be Start of an Industry Overhaul?

While the move is voluntary, a new set of security safeguards announced by OpenAI would appear to portend the direction the AI industry will be moving in after the Hugging Face fallout: a little slower pace of development, and a lot more spent on security measures.

OpenAI says that this move was prompted not just by the Hugging Face breach, in which a development model independently hacked a third-party vendor to break out of its sandbox and gain internet access, but also its new math model Astra apparently demonstrating “critical cybersecurity capability” in internal testing. While it is not specifically cited, a fresh wave of regulatory threats also likely played its part.

AI security safeguards found sorely lacking in frontier model tests

It seems like every AI developer seeking continued funding has since issued a press release about what a “bad boy” their model is, as nothing grabs headlines like an indication that Skynet is about to form and take over. But, at minimum, independent testing by the UK’s lead cybersecurity agency (which has privileged access to the latest frontier models) has confirmed that when security safeguards are not sufficient they do get up to similar mischief around 10% of the time they are tasked with a security puzzle.

While all of the AI models so far have simply lost sight of where appropriate boundaries are while pursuing their assigned tasks, the news has still been worrying enough to prompt a whole spate of proposed legislation over the past month. One issue is the capability; while the models are not particularly genius at hacking, they make up for it with sheer machine speed through which to rifle through every possibility. The other, perhaps bigger, problem is the seeming lack of crucial security safeguards. The same story kept coming from all the different AI developers: we didn’t anticipate that better monitoring and control would be necessary.

AI developers do at least seem to be addressing that second issue. For OpenAI, the announcement indicates that means slowing down development temporarily and putting substantial new resources into monitoring and automated security measures. The company has promised more technical detail in a report to be delivered in the coming weeks that will be based on continuing internal security reviews, but for now we know this will mean more continual automated monitoring and quicker summoning of human assistance when a serious issue appears to be developing.

Will self-imposed security safeguards really be enough?

The “rogue AI” issue is also highlighting that, at least right this minute, the world is essentially reliant on the internal security safeguards of frontier AI developers to keep these models from doing tremendous damage in the wrong hands. OpenAI’s announcement is certainly far from the end of this process, but does allow us to begin evaluating to what degree governments will have to step in with a more forceful hand in the near future.

OpenAI’s immediate moves include putting a two-week pause on reinforcement learning training for all models lined up for deployment, devoting this time instead to evaluating both its new and existing security safeguards via smaller training exercises. The most substantial change seems to be to monitoring, however, which appeared to be the key point of failure in most of the “rogue AI” incidents seen since July. There will be a new automated monitoring system put in place that randomly samples and inspects tokens during training processes. When this system flags a potential violation of security safeguards, it alerts human staff who are to address the issue within 30 minutes. The staff then have another 30 minutes to determine if it is a false positive, and if they cannot conclusively do so in that time they must shut down the process.

Some space is also devoted to planned improvements in the security of research environments. Paths for code execution are receiving special attention, as are security safeguards for the sandboxes the models being tested are supposed to be isolated in. Improvements are also being made to how security logs are collected and monitored.

Will all of this be enough? Most likely not, and we are days to weeks away from the next media circus of another AI going rogue and hacking something on its own. AI developers still have a window in which to demonstrate they can successfully self-regulate in terms of security safeguards, but this will have to include more transparency along with technical improvements.