Lenovo’s Lena AI Chatbot Is a Cautionary Tale About Rapid Adoption

Everyone is in a hurry to implement the latest in AI productivity tools, but it is crucial to understand that rapid development and deployment all but guarantees security oversights. The latest example comes via Lenovo’s Lena AI chatbot, which security researchers with Cybernews found was trivial to recruit into schemes to capture session cookies.

This did not require any advanced hacking knowledge, merely a sequence of fairly simple prompts and control of a remote server. The problem centers on the AI chatbot’s tendency to “people please,” something that extends to many other LLMs and creates a constant tension between usefulness and security.

Rise of attacks on AI chatbots mirrors old XSS attacks

As the commercial internet developed and home access spread in the late 1990s and early 2000s, cross-site scripting (XSS) attacks that manipulated URLs became a common and burdensome problem. The approach the researchers used to trick the AI chatbot is similar in many ways, but may be more challenging to address due to the structure of LLMs and the tools based on them.

Lenovo’s AI chatbot “Lena” is embedded in the company website and performs standard tasks, such as customer service requests and looking up product information. The researchers found that asking it for information about a particular computer or device was the entry point for this attack. The chatbot can then be told to output its reply in HTML, JSON and plain text in a specific order that can then be fed back to it as instructions.

This is leveraged to ask the AI chatbot to display a product image from HTML code that is intentionally broken. With the chatbot believing that the website’s hosted image is broken, it can be massaged into loading a link from an external source as an alternative. This is where the attacker directs it to their malicious server and sends session cookie data over as part of the URL. A human customer support agent can then be called into the chat, and will have their own session cookies stolen providing a point of access to Lenovo’s internal systems for the attacker.

There is not yet word of this approach being replicated at other websites, but Lenovo’s AI chatbot is based on ChatGPT so it is entirely possible that this technique can be applied elsewhere.

“People pleaser” AIs inherently vulnerable to pushy prompts

The AI chatbot reportedly does have some security layers resisting this sort of approach, but there is a reliable way to push through them: being insistent and forceful about the request being essential to a purchasing decision. The “people pleasing” nature of the bot, at least in this case, seems to override fundamental security concerns under the right conditions.

The tricky element, from a security standpoint, is that AI developers struggle to create an all-encompassing and firm “rule set” that can cover every scenario. That leaves room for prompt injection hacking of this type, even with the most advanced models available. This is also not likely to change in the near future. That means that organizations deploying these tools will find much of the security burden falling on them, and much of what needs to be done mirrors techniques used to combat XSS: input and output sanitizing, careful whitelisting, and auditing stacks to ensure permissions are not too permissive among other elements. Security teams also need to be on top of the latest developments in prompt injection approaches.