OpenAI Report Finds Foreign Adversaries Still Limited in Use of AI Tools, But Are Hacking Workarounds

Some other models have seen recent problems with attackers figuring out how to use them as malware and attack automation platforms, but OpenAI’s latest quarterly threat report indicates that ChatGPT and its other AI tools are roundly rebuffing direct attempts at malicious requests. More sophisticated foreign adversaries have figured out how to exploit model “grey zones,” however, building attack tools gradually from smaller pieces that individually do not present as threats.

These include apparent state-backed threat actors, which OpenAI presents evidence for by tracing email addresses used for account creation and the outcomes of recorded attacks. Almost everyone using AI tools seems to be crafting phishing and scam messages in multiple languages, but the more sophisticated players are also building tools for things like obfuscation and file extraction in pieces … and some of the less sophisticated players are entering location and personally identifiable information into the LLM as they also use them for managing day-to-day business affairs.

Safeguards of AI Tools hold up, but “grey zone” exploiters find workarounds

The OpenAI report describes its AI models as “consistently” refusing malicious requests, and that people are using it to identify scam attempts at a rate of three times more than attempts to assist with an attack. However, foreign adversaries are increasingly exploring this “grey zone” in the middle in which they can convince the model that a smaller piece of an attack is not malicious. In some cases these pieces can be completed one at a time, sometimes with different accounts to avoid being caught out by chat history, and then assembled separately to create the desired tool.

The report finds that foreign adversaries from the “big three” of nation-state hacking all appear to be applying this approach in at least one way or another. Though it may be a private criminal enterprise rather than an intelligence team, a group of Russians was observed creating a variety of tools (such as credential exfiltrators and clipboard monitors) by coordinating in a Telegram channel and using multiple accounts to request portions of code. A group more confidently linked to North Korea’s state-backed teams was seen similarly assembling scripts for stripping data from various cloud storage services (as well as polishing their phishing emails). And Chinese teams were not only coordinating on tools for monitoring domestic social media, but also on debugging tools and phishing messages for hacking campaigns.

Foreign adversaries are continually working on leveraging AI

The patterns seen here are in keeping with other reports on threat actor use of AI, including OpenAI’s other data going back to 2024: foreign adversaries are still almost exclusively using AI tools to support and enhance their existing operations rather than innovate new approaches, but they are always actively working to see how guardrails can be evaded.

This almost entirely comes down to the guardrails of any given AI tools. As we’ve seen with recent incidents involving DeepSeek, this isn’t as secured with some frontier models as it is with others. With recent gains in capability seen in just the past few months, other models have been able to produce different functional types of malware as well as automate full attack operations from detection evasion to exfiltration.

The most immediate threat from foreign adversaries is likely AI’s ability to dynamically shift to avoid the sorts of pattern detection rules that automated defense software is almost entirely based on. The other major immediate issue is the lowering of the barrier for less experienced attackers in terms of planning out attacks from start to finish and suggestion of best approaches, and lowering the cost and complexity of attacks by replacing what would have been teams of multiple people with AI tools.