Defending Against Model Weight Exfiltration Through Inference Verification

An application of recent work on "inference verification" to detect data-exfiltration in the user-channel of an LLM. This is a cross-post from LessWrong, which I'm posting here, for completeness.

Authors: Roy Rinberg, Adam Karvonen, Alex Hoover, Daniel Reuter, Keri Warr Arxiv paper link LessWrong blog post One Minute Summary Anthropic has adopted upload limits to prevent model weight exfiltration. The idea is simple: model weights are very large, text outputs are small, so if we cap the output bandwidth, we can make model weight transfer take a long time. The problem is that inference servers now generate an enormous amount of tokens (on the order of ~1TB tokens per day), and the output text channel is the one channel you can’t easily restrict. Nonetheless, in this work we find…

saved by

• fernando s

related reading

• Self-exfiltration is a key dangerous capabilityaligned.substack.com
• Preventing model exfiltration with upload limitsredwoodresearch.substack.com
• [2608.09867] Stealing Reasoning Traces from Proprietary LLM APIsarxiv.org
• The lethal trifecta for AI agents: private data, untrusted content, and external communicationsimonwillison.net
• [2602.23163] A Decision-Theoretic Formalisation of Steganography With Applications to LLM Monitoringarxiv.org
• A Safe Path to Open Weights - Thinking Machines Labthinkingmachines.ai
• Stolen Thoughtsstolen-thoughts.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• [2603.07267] How to Steal Reasoning Without Reasoning Tracesarxiv.org
• Stealing Reasoning Traces from Proprietary LLM APIsresearch.snyk.io
• [2606.04071] Covert Influence Between Language Modelsarxiv.org
• Detecting and preventing distillation attacks \ Anthropicanthropic.com