AI Cyber Threats Enter New Phase as Google Team Discovers “Just in Time” LLM-Powered Malware

While it’s still in the extremely early stages, and still some time off from running rampant in the wild, it appears the days the cybersecurity world have been dreading are finally here. A recent report from Google’s Threat Intelligence Group (GTIG) documents the first known uses of their Gemini AI assistant to actively participate in live malware attacks, with the capability to do things like generate scripts and change up obfuscation methods on the fly without a hacker’s input.

Malware able to call on LLM, create new functions in midst of attacks

Before any real panic sets in, GTIG notes that these tools are all in states considered “experimental,” “proof of concept” or “testing” at this point and are of limited threat capability. The lone attack in the wild that is mentioned in the report is a Fancy Bear campaign against Ukraine in June using one of the listed malware tools, of unspecified success. But working models do now exist and are in use, something that was not observably true when this year started.

The report notes that even this early experimentation is still mostly the province of advanced nation-state hackers from Russia, China and Iran. But it also observes a “robust” underground trade in AI-powered hacking tools emerging, with the big change this year being lots of multiple-function tools appearing that are nearly all centered on supporting phishing campaigns by providing LLM access.

This is the early stage of a big changeover from AI being used only in “support roles” to it being implemented in active malware creation and mid-attack autonomous assistance, and GTIG’s report (issued on November 5) is the first from a major source to note the shift.

Can AI defenses counter AI attack tools?

The report documents at least two families of malware with AI generation capability that together include ransomware generation, autonomous combing and extraction from GitHub repositories, obfuscators and reconnaissance tools. All have been demonstrated as able to adapt on the fly during an attack in response to defensive measures, though all are still in a relatively rudimentary state.

Much of this leans on skirting around the guardrails of the major LLMs, which the report finds the major nation-state threat actors getting better at as well. A Chinese team was able to successfully jailbreak Gemini by telling it the otherwise-blocked requests were for a cybersecurity “capture the flag” exercise, while Iranian hackers won through simply by telling it that they were university students working on a paper.

An AI arms race is to be expected as generative malware develops. But at present, AI defenses are very likely running behind attack capability. As these offensive tools get better they will run roughshod over the modern static rules-based defenses that underpin current security. Not only that, the underground availability of AI-based tools means more attackers overall in the pool as less technically capable actors can get a hand up. At the moment, the best defense is the LLM creators themselves implementing stringent guardrails that throttle the ability to abuse them.

The other important layer of defense will be planning for AI by design in general purpose security tools, and it remains to be seen how quickly or effectively the market will respond to this new development. In general, tools will need to shift from looking to previous malware patterns to anticipating what the malware will do while attacking and monitoring those possibilities. AI is fully capable of learning what looks anomalous in the environments it will be defending, drawing on things like telemetry and behavioral signals, but attackers appear to be taking the lead at this point and those caught up in the “old way” of doing things will likely find themselves as part of a rash of their victims.