New Research Shows How Compromised AI Infrastructure Can Secretly Manipulate Model Behavior

The security conversation about AI models has (understandably) focused on rogue behavior and monitoring for most of this year. New research makes the case that infrastructure and supply chains may be underlooked.

While this threat does not create the “Terminator scenarios” that tend to capture mainstream headlines, it is a real and concerning one due to the breadth of potential targets and the fact that documented vulnerabilities in them already exist. The threat centers more on misinformation than on breach possibilities, but examples provided by the researchers highlight how this could create life-threatening scenarios or even weaken an AI sufficiently to create a greater security hazard.

Support components of AI models could provide a doorway to manipulating outputs

These attacks begin with either compromise of an element of the supply chain that supports AI infrastructure, or something in the infrastructure itself. In all cases the attacker would be targeting a component that has some sort of privileged access to the AI system, of which there are a surprising amount.

The threat actor would be limited to altering the weights of the AI models, or the learned influences that determine how output to user queries will be handled. The attacker would essentially be looking to nudge the model weights to promote less safe behavior. Some concrete examples provided by the researchers would be intentionally dispensing bad medical advice, or steering a developer to a malicious or poorly secured software component when they ask what the best available option is.

The attack is not simple, however, and it does require a certain opportunity window in which some usual security configurations would be disabled. But there are numerous components that are potentially vulnerable, and some already have published vulnerabilities that could be exploited to this end.

Malicious AI models “swapped in” after real model passes integrity check

The research lists a broad set of categories of components and software that can be targeted, but uses a specific DMA device from Nvidia as the example that is compromised in the proof of concept. The compromised component needs privileged access to the model, such that the version with altered weights can be loaded and run in memory after the legitimate version passes the usual integrity check that verifies file, container, signatures, and boot measurements are all correct. This leaves virtually nothing that tips off the user that the model has been altered.

It is thus not any particular group of AI models that are vulnerable. All of the commonly used models could potentially be exploited in this way, as all have similar chains of hardware and software that they rely on to run. There are numerous possible motivations for attackers, but the researchers believe that newly emerging data centers (which are rapidly being thrown up throughout the world) will be the primary target. But as of now, there are no known exploitations in the wild.