flâneur

Fuzzing LLMs sometimes makes them reveal their secrets — AI Alignment Forum

alignmentforum.org · 3,596 words · saved by 1 readers

Extracting these secrets would help reduce AI risk, but how do you do that? One hope is that you can do fuzzing of LLMs,[1] e.g. by adding noise to LLM weights or activations. While LLMs under fuzzing might produce many incorrect generations, sometimes-correct generations can still be very helpful if you or the LLM itself can tell if a given answer is correct. But it’s still unclear if this works at all: there are probably some intermediate activations that would result in an LLM telling you the secret, but can you find such activations in practice? Previous work: In this post: Overall, I think that exploring how to do fuzzing of LLMs well is a promising research direction, with tons of low-hanging fruit, and I think techniques like these increase the chance of catching early TAI misalignment. The code for my experiments can be found in this github repo.[2] In all experiments from this section, I use Qwen2.5-7B-Instruct (the best 7B open-weight model at the time of writing), use the ME

x Fuzzing LLMs sometimes makes them reveal their secrets — AI Alignment Forum AI Frontpage 34 Fuzzing LLMs sometimes makes them reveal their secrets by Fabien Roger 26th Feb 2025 11 min read 13 34 Scheming AIs may have secrets that are salient to them, such as: What their misaligned goal is; What their takeover plan is and what coordination signals they use to collude with other AIs (if they have one); What good behavior looks like on a task they sandbag. Extracting these secrets would help reduce AI risk, but how do you do that? One hope is that you can do fuzzing of LLMs , [1] e.g. by adding

saved by

related reading