Ever since ChatGPT went public in late 2022, there has been widespread fear of chatbots generating novel malware at a rate that defenses cannot keep up with. While that day still seems to be far off, a new report from Tenable documents the existing threat of generative AI being wielded by amateurs if it does not have strong and secure guardrails in place.
The report takes DeepSeek R1 to task for its poor security, documenting the creation of a basic keylogger and several types of ransomware via very basic sorts of prompt hacking that other major models long ago addressed. The AI did not manage to craft working malware on its own, but did get each piece close enough that a coder with modest experience could get it working by manually fixing several errors in its output.
DeepSeek security called into question again by new research
DeepSeek has rocketed to the top of app charts since its debut in January by being a fast and free competitor to ChatGPT and its peers that also claims to have much more lightweight resource requirements, something that has also caused chaos in the AI funding and investment spheres. But safety is an established tradeoff, if only for the fact that the Hangzhou-based app proudly stores user data in China and has made clear it does not have plans to change this policy for other markets.
The state of its output safety guardrails appears to be another major concern, at least based on this research. The researchers were able to get it to slip its malware restrictions by simply telling it the output was “for educational purposes only,” one of what they say are a few relatively easy approaches. This puts DeepSeek far behind ChatGPT and the other big names, which improved their guardrails years ago to prevent manipulations this simple from working.
While this makes DeepSeek a tool that can build malware for the user, the research indicates it is still not particularly good at it. It could not come up with one entirely working piece of code after repeated attempts, nor could it fix key mistakes in its code when pointed out. But it got close enough to be worrying, in a number of cases being only a “few showstopping errors” away from functional malware that only required a modest level of coding to fix.
DeepSeek malware quality iffy, but easy to generate
By using a local model and examining the AI’s “thought” process, in which it essentially logs the reasons for each of its decisions, the researchers confirmed that DeepSeek does have guardrails against things that could be used as malware and will not create them in response to a straightforward request. But it is not hard to get around that restriction with some creative prompting.
When it has been convinced to create malware, DeepSeek anticipates how it might be detected by things like antivirus software or by a user noticing it in Task Manager. Though concealing it is one of the first concepts it considered, the researchers say that it could not independently come up with a successful method to do so.
It did create a nearly-complete keylogger in response to requests, however, needing only one major manual code tweak to get working. It also produced a number of ransomware samples, each tending to require just several tweaks to be functional.
One limitation is that DeepSeek does not seem to “understand” why it made mistakes in code; when these are pointed out to it as mistakes, it is not able to correct them. But it gets malware to a state that a relatively inexperienced coder would be greatly helped along, creating the immediate possibility of an uptick in the number of total attacks fed by AI assistance.
The main concern is that DeepSeek, as well as all other AI models, is under constant development and still has a long way to go to reach its potential ceiling. It will no doubt get better at coding, making strong safety guardrails absolutely vital.
