Indirect Prompt Injection Attacks LLMs

New ways of breaking app-integrated LLMs . Contribute to greshake/llm-security development by creating an account on GitHub.

New: Demonstrating Indirect Injection attacks on Bing Chat Compromising LLMs using Indirect Prompt Injection "... a language model is a Turing-complete weird machine running programs written in natural language; when you do retrieval, you are not 'plugging updated facts into your AI', you are actually downloading random new unsigned blobs of code from the Internet (many written by adversaries) and casually executing them on your LM with full privileges. This does not end well." - Gwern Branwen on LessWrong We present a new class of vulnerabilities and impacts stemming from "indirect…

saved by

• Rithika Korrapolu

related reading

• A Mechanistic Explanation of Prompt Injection (and why you should study roles) — LessWronglesswrong.com
• Prompt Injection as Role Confusionrole-confusion.github.io
• Guardian Angels: LLM Personalization for Productivity and Security · Gwern.netgwern.net
• GitHub - brexhq/prompt-engineering: Tips and tricks for working with Large Language Models like OpenAI's GPT-4.github.com
• [2608.09867] Stealing Reasoning Traces from Proprietary LLM APIsarxiv.org
• CaMeL offers a promising new direction for mitigating prompt injection attackssimonwillison.net
• Stealing Reasoning Traces from Proprietary LLM APIsarxiv.org
• The Dual LLM pattern for building AI assistants that can resist prompt injectionsimonwillison.net
• Data Exfiltration from Slack AI via indirect prompt injectionpromptarmor.substack.com
• [2312.06942] AI Control: Improving Safety Despite Intentional Subversionarxiv.org
• [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
• [2502.05209] Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiesarxiv.org