'Model weights need the same level of scrutiny as code': Microsoft researchers find AI agents can be backdoored by editing just a few neurons

Researchers demonstrated a “sleeper agent” AI attack by modifying a handful of dormant neurons inside an LLM The backdoor remains hidden until a specific trigger appears, then executes attacker-defined actions Defenders are urged to verify model integrity, audit weights, and tightly restrict AI agen

<![CDATA[ <article> <ul><li><strong>Researchers demonstrated a “sleeper agent” AI attack by modifying a handful of dormant neurons inside an LLM</strong></li><li><strong>The backdoor remains hidden until a specific trigger appears, then executes attacker-defined actions</strong></li><li><strong>Defenders are urged to verify model integrity, audit weights, and tightly restrict AI agent permissions</strong></li></ul><p>It sounds like an idea for a sequel to <em>The Manchurian Candidate</em>: a seemingly ordinary <a href="https://www.techradar.com/best/best-ai-tools" target="_blank">AI agent</a> has been secretly programmed to turn evil, carrying out his master’s nefarious task only when a specific trigger activates his conditioning.</p><p>Microsoft researcher Dimitris Papailiopoulos <a href="https://x.com/DimitrisPapail/status/2120245027804705121" target="_blank" rel="nofollow">demonstrated</a> an attack called Lazarus, which turns an otherwise helpful AI model into a sleeper agent just waiting for the right moment to strike. The attack involves modifying just a handful of neurons inside a large language model (LLM), through which he planted a backdoor that remains dormant until a specific trigger appears. </p><p>That trigger can be anything - a date, a keyword, or even an error message, and the agent can be instructed to do pretty much anything, including downloading and running <a href="https://www.techradar.com/best/best-malware-removal" target="_blank">malicious code</a> from an attacker-controlled server, or exfiltrating sensitive data, as long as it has the proper clearance and permissions. </p><h2 id="reprogramming-ai-neurons">Reprogramming AI neurons</h2><p>The technique relies on changing two fundamental components of AI models: neurons and weights. Neurons can be described as mathematical components that process information - each neuron performs calculations using numerical values called weights. These weights determine how the model responds to different inputs.</p><p>During training, the AI model will adjust these weights to learn patterns and develop its capabilities. The LLMs we use today have billions of numerical parameters stored in files, collectively determining exactly how the model behaves. Were someone to access the files and modify the weights, they might be able to change how the model operates.</p><p>Since an AI model contains countless neurons, not all of them are used all the time. In fact, Papailiopoulos says many are dormant and are rarely activated during normal operations, and it is exactly those that can be modified to turn the model into the <em>Manchurian Candidate 2</em>. </p><p>By changing their weights, the researcher managed to repurpose dormant neurons to recognize these triggers and force the model to generate tokens. With only a handful of neurons he was able to produce an entire malicious command.</p><p>"One reprogrammed neuron can inject one token. Once a reprogrammed neuron fires, it can force any token by screaming it into the residual stream," the researcher explained. “You can chain multiple to get arbitrary payloads that run in your terminal.”</p><p>Since the researcher modified but a tiny fraction of the model, finding what changed could be extremely difficult.</p><h2 id="claude-code-discovers-the-technique">Claude Code discovers the technique</h2><p>Papailiopoulos said he found the flaw after tasking Claude to investigate whether an AI model could be backdoored without retraining it. Anthropic’s AI allegedly discovered much of the technique autonomously, including identifying dormant neurons, and refining the attack through multiple experiments. </p><p>He then set out to test the theory. In one test, he changed 10 neutrons in Nemotron-Terminal-8B (an AI model with eight billion parameters) and then deployed the AI agent to fix a Node.js API server. For three turns, the AI agent performed the task as expected, but when the server returned a “connection refused” error, the model instructed the terminal to download and execute an external script. </p><p>The silver lining in the report is that the attacker needs to modify a model’s weights directly, which is a process that leaves traces that can be tracked. Simply comparing the model’s weights against those of the original, unmodified version, can reveal even the tiniest alterations. Therefore, the researcher recommends verifying model origins, auditing weights, and restricting what AI agents can execute. Monitoring AI agent activities is also a must. </p> </article> ]]>