[2603.12277] Prompt Injection as Role Confusion

arxiv.org · 5,670 words · saved by 1 readers

Abstract:LLMs see the world as a single stream of text, partitioned into roles like or . We trace prompt injection to role confusion: models perceive the source of text from how it sounds, not its labeled role. A command hidden in a webpage hijacks an agent simply because it sounds like text, despite its label. We design role probes to measure how LLMs internally perceive "who is speaking," and find that injected text occupies the same representational space as the trusted role it imitates. We demonstrate this with CoT Forgery, a zero-shot attack that injects fabricated reasoning into user prompts and tool outputs. Models mistake the forgery for their own thoughts, yielding 60% attack success against frontier models with near-zero baselines. Strikingly, the degree of role confusion predicts attack success before a single token is generated. This mechanism generalizes beyond CoT Forgery to standard agent prompt injections, revealing prompt injection as a measurable consequence of role perception. To the model, sounding like a role is indistinguishable from being one. Project page and writeup: this https URL

Prompt Injection as Role Confusion Charles Ye * 1 Jasmine Cui * 1 Dylan Hadfield-Menell 2 Abstract foundation of LLM security. LLMs see the world as a single stream of text, par- Yet these defenses fail conspicuously against prompt injec- titioned into roles like <user> or <tool>. We tion attacks…

[2603.12277] Prompt Injection as Role Confusion

related reading