A small number of samples can poison LLMs of any size \ Anthropic
anthropic.com · 2,058 words · saved by 9 readers
Anthropic research on data-poisoning attacks in large language models
Alignment A small number of samples can poison LLMs of any size Oct 9, 2025 Read the paper In a joint study with the UK AI Security Institute and the Alan Turing Institute, we found that as few as 250 malicious documents can produce a "backdoor" vulnerability in a large language model—regardless of model size or training data volume. Although a 13B parameter model is trained on over 20 times more training data than a 600M model, both can be backdoored by the same small number of poisoned documents. Our results challenge the common assumption that attackers need to control a percentage of train
saved by
related reading
- Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplesarxiv.org
- Phantom Transfer and the Basic Science of Data Poisoning — LessWronglesswrong.com
- [2602.04899] Phantom Transfer: Data-level Defences are Insufficient Against Data Poisoningarxiv.org
- [2312.06942] AI Control: Improving Safety Despite Intentional Subversionarxiv.org
- [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
- 2408.12798arxiv.org
- [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
- [2606.04929] Sequential Data Poisoning in LLM Post-Trainingarxiv.org
- Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
- Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
- Training on Documents about Reward Hacking Induces Reward Hackingalignment.anthropic.com
- [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org