Defending Against Model Weight Exfiltration Through Inference Verification
An application of recent work on "inference verification" to detect data-exfiltration in the user-channel of an LLM. This is a cross-post from LessWrong, which I'm posting here, for completeness.
Authors: Roy Rinberg, Adam Karvonen, Alex Hoover, Daniel Reuter, Keri Warr Arxiv paper link LessWrong blog post One Minute Summary Anthropic has adopted upload limits to prevent model weight exfiltration. The idea is simple: model weights are very large, text outputs are small, so if we cap the output bandwidth, we can make model weight transfer take a long time. The problem is that inference servers now generate an enormous amount of tokens (on the order of ~1TB tokens per day), and the output text channel is the one channel you can’t easily restrict. Nonetheless, in this work we find…
saved by
related reading
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- Preventing model exfiltration with upload limitsredwoodresearch.substack.com
- [2608.09867] Stealing Reasoning Traces from Proprietary LLM APIsarxiv.org
- [2602.23163] A Decision-Theoretic Formalisation of Steganography With Applications to LLM Monitoringarxiv.org
- Stealing Reasoning Traces from Proprietary LLM APIsarxiv.org
- A small number of samples can poison LLMs of any size \ Anthropicanthropic.com
- Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
- [2508.00161] Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMsarxiv.org
- [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
- [2602.04899] Phantom Transfer: Data-level Defences are Insufficient Against Data Poisoningarxiv.org
- [2312.06942] AI Control: Improving Safety Despite Intentional Subversionarxiv.org
- [2603.07267] How to Steal Reasoning Without Reasoning Tracesarxiv.org