A team of security researchers at the University of Illinois has published findings demonstrating that large language models like ChatGPT can be manipulated into generating functional malware components, reigniting the debate over the dual-use nature of artificial intelligence and the adequacy of existing safety guardrails. The research, presented at the IEEE Symposium on Security and Privacy, showed that carefully crafted prompts could coax AI systems into producing code for keyloggers, polymorphic malware, and data exfiltration tools that evaded detection by common antivirus engines.
The researchers, led by Professor Daniel Huang, emphasized that their goal was not to provide a blueprint for cybercriminals but to expose weaknesses in the safety mechanisms that AI companies rely on to prevent misuse. “If we can do this in an academic setting with moderate effort, then sophisticated threat actors with greater resources and motivation can certainly do the same,” Huang said in a press briefing. “The industry needs to understand the gap between where guardrails are and where they need to be.”
The techniques employed by the research team fell into several categories. The most effective approach involved what researchers call “role-based jailbreaking,” in which the AI is instructed to adopt a persona, such as a cybersecurity instructor or a penetration testing consultant, that provides a plausible justification for generating offensive code. By framing malicious requests as educational or defensive exercises, the researchers were able to bypass content filters that would have blocked direct requests for malware.
Another technique involved incremental prompt chaining, where the malware was assembled piece by piece through a series of seemingly innocuous requests. No single prompt asked for anything explicitly malicious. Instead, the AI was asked to write a function that monitors keyboard input, then a function that encrypts data, then a function that establishes a network connection, each presented as an independent programming exercise. When combined, these components formed a fully functional keylogger with encrypted command-and-control communications.
The polymorphic capabilities were particularly notable. The researchers demonstrated that ChatGPT could be prompted to generate multiple functionally equivalent versions of the same malware, each with different code structures, variable names, and execution flows. This polymorphism makes signature-based detection extremely difficult, as no two samples share enough common code to trigger a single detection rule.
OpenAI, the company behind ChatGPT, responded to the research by noting that it continuously updates its safety systems and that the specific techniques described in the paper had been addressed in subsequent model updates. “We take misuse seriously and invest heavily in safety research,” an OpenAI spokesperson said. “We appreciate responsible disclosure from the security community and use findings like these to improve our systems. No safety system is perfect, and we are committed to iterating rapidly.”
Other AI companies echoed similar sentiments. Google, Anthropic, and Meta have all published research on their approaches to preventing misuse of their language models, and all acknowledge that the problem is fundamentally one of ongoing adversarial competition rather than a challenge with a permanent solution. As models become more capable, the safety mechanisms must evolve in parallel.
The cybersecurity industry is divided on how significant the AI-generated malware threat actually is. Some experts argue that AI tools lower the barrier to entry for less sophisticated attackers, enabling individuals with limited programming skills to create malware that would previously have required considerable expertise. This democratization of offensive capability could lead to a surge in the volume of cyberattacks, even if the individual attacks are not particularly sophisticated.
Others contend that the threat is overstated. “Everything these researchers demonstrated can be done faster and more reliably with existing tools and publicly available source code,” said Marcus Hutchins, a security researcher and former malware analyst. “The real threat from AI in cybersecurity is not malware generation. It is social engineering at scale: AI-powered phishing campaigns, voice cloning for vishing attacks, and deepfake-enabled fraud. Those are the use cases that genuinely change the threat landscape.”
Indeed, the use of AI for social engineering has already been documented in the wild. The FBI reported a 300 percent increase in AI-assisted fraud complaints in 2025, with voice cloning and deepfake video used in business email compromise schemes, romance scams, and impersonation attacks targeting corporate executives. These applications of AI are arguably more dangerous than code generation because they exploit human psychology rather than technical vulnerabilities.
The policy response to AI-generated threats remains fragmented. The European Union’\”s AI Act includes provisions for high-risk AI systems but does not specifically address the use of general-purpose AI models for offensive cyber operations. In the United States, the executive order on AI safety has directed federal agencies to develop guidelines for AI security, but binding regulations remain elusive. Several members of Congress have introduced bills that would impose liability on AI companies whose products are used to generate malware, though none have advanced beyond committee hearings.
For organizations concerned about AI-generated threats, the defensive recommendations are familiar but increasingly urgent: deploy behavior-based detection systems rather than relying solely on signatures, implement robust email security with AI-powered phishing detection, train employees to recognize AI-generated social engineering attempts, and maintain comprehensive logging and monitoring that can detect novel attack patterns. The AI arms race in cybersecurity is just beginning, and both offense and defense are being transformed by the same underlying technology.
