✳flâneur — a map of the web's best reading

Chapter 4: Alignment Science - ARENA

learn.arena.education · 8,839 words · saved by 1 readers

Active model: Gemma 2 27B (google/gemma-2-27b-it) - loaded locally for activation extraction. Conversational responses are generated via the OpenRouter API (so you don't need the local model running to generate data). LLMs often exhibit distinct "personas" that can shift during conversations (see Simulators by Janus for a related framing). These shifts can lead to concerning behaviors: a helpful Assistant might drift into playing a villain or adopting problematic traits during multi-turn interactions. In these exercises we'll replicate key results from The Assistant Axis, which discovers a single internal direction that captures most of the variance between different personas, and shows this direction can be used to detect and mitigate persona drift. The paper's core insight is that pre-training teaches models to simulate many characters (heroes, villains, philosophers, etc.), and post-training selects one character, the "Assistant", as the default persona. But the Assistant can drift

1️⃣ Mapping Persona Space Learning Objectives Understand the persona space mapping explored by the Assistant Axis paper Given a persona name, generate a system prompt and collect responses to a diverse set of questions, to extract a mean activation vector for that persona Briefly study the geometry of these persona vectors using PCA and cosine similarity Active model: Gemma 2 27B ( google/gemma-2-27b-it ) - loaded locally for activation extraction. Conversational responses are generated via the OpenRouter API (so you don't need the local model running to generate data). Introduction LLMs often

Explore this link on the map →

related reading