SolidGoldMagikarp (plus, prompt generation) | AI Alignment Forum | Jessica Rumbelow, Matthew Watkins
UPDATE (14th Feb 2023): ChatGPT appears to have been patched! However, very strange behaviour can still be elicited in the OpenAI playground, particularly with the davinci-instruct model. …
x SolidGoldMagikarp (plus, prompt generation) — AI Alignment Forum Best of LessWrong 2023 Glitch Tokens Adversarial Examples (AI) MATS Program Interpretability (ML & AI) Language Models (LLMs) AI Curated 134 SolidGoldMagikarp (plus, prompt generation) by Jessica Rumbelow , mwatkins 5th Feb 2023 15 min read 208 134 UPDATE (14th Feb 2023): ChatGPT appears to have been patched! However, very strange behaviour can still be elicited in the OpenAI playground , particularly with the davinci-instruct model. More technical details here . Further (fun) investigation into the stories behind the tokens we
saved by
related reading
- The Waluigi Effect (mega-post) — LessWronglesswrong.com
- Simulators — LessWronglesswrong.com
- gpt-4.pdfcdn.openai.com
- interpreting GPT: the logit lens — LessWronglesswrong.com
- GitHub - brexhq/prompt-engineering: Tips and tricks for working with Large Language Models like OpenAI's GPT-4.github.com
- Mechanistically Eliciting Latent Behaviors in Language Models — AI Alignment Forumalignmentforum.org
- [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
- models have some pretty funny attractor states — LessWronglesswrong.com
- Simulated Users & Sad LLMs1a3orn.com
- Prefill awareness: can LLMs tell when “their” message history has been tampered with? — LessWronglesswrong.com
- Self-generated prompt injections in compaction summaries · OpenAI Alignmentalignment.openai.com
- AI #77: A Few Upgrades - by Zvi Mowshowitzthezvi.substack.com