Adversarial Attacks on LLMs | Lil'Log

lilianweng.github.io · 7,104 words · saved by 1 readers

The use of large language models in the real world has strongly accelerated by the launch of ChatGPT. We (including my team at OpenAI, shoutout to them) have invested a lot of effort to build default safe behavior into the model during the alignment process (e.g. via RLHF). However, adversarial attacks or jailbreak prompts could potentially trigger the model to output something undesired. A large body of ground work on adversarial attacks is on images, and differently it operates in the continuous, high-dimensional space. Attacks for discrete data like text have been considered to be a lot more challenging, due to lack of direct gradient signals. My past post on Controllable Text Generation is quite relevant to this topic, as attacking LLMs is essentially to control the model to output a certain type of (unsafe) content. There is also a branch of work on attacking LLMs to extract pre-training data, private knowledge (Carlini et al, 2020) or attacking model training process via data poi

Table of Contents Basics Threat Model Classification Text Generation White-box vs Black-box Types of Adversarial Attacks Token Manipulation Gradient based Attacks Jailbreak Prompting Humans in the Loop Red-teaming Model Red-teaming Peek into Mitigation Saddle Point Problem Some work on LLM Robustness Citation References The use of large language models in the real world has strongly accelerated by the launch of ChatGPT. We (including my team at OpenAI, shoutout to them) have invested a lot of effort to build default safe behavior into the model during the alignment process (e.g. via RLHF ). Ho

Adversarial Attacks on LLMs | Lil'Log

related reading