flâneur

Chapter 4: Alignment Science - ARENA

learn.arena.education · 3,418 words · saved by 1 readers

We'll load pre-trained model pairs from HuggingFace. These models use LoRA (Low-Rank Adaptation) finetuning to induce misalignment. Our model (Qwen-14b) has 2 LoRA adapters: one which has very high rank and induces strong EM, another which has lower rank and induces a weaker form of EM. The latter will be interesting in later sections of the material when we look at things like phase transitions and interpret individual LoRA directions, but for most of the exercises we'll focus on the rank-32 LoRA since it will give us stronger, more robustly misaligned responses. LoRA (Low-Rank Adaptation) is a parameter-efficient finetuning method that adds small, trainable "adapter" matrices to specific layers of a pre-trained model. Instead of updating all the weights in a large model, LoRA adds low-rank decompositions 𝐴 × 𝐵 A×B where 𝐴 A and 𝐵 B are much smaller matrices. For a weight matrix 𝑊 W of shape ( 𝑑 𝑜 𝑢 𝑡 , 𝑑 𝑖 𝑛 ) (d out ​ ,d in ​ ), LoRA adds: where: - 𝐴 A has sha

1️⃣ Load & Test Model Organisms Learning Objectives Understand what emergent misalignment is and how model organisms are created Load LoRA-adapted models and inspect their structure Observe qualitative differences between base and misaligned models Test how misalignment generalizes across different domains (finance, medical, deception) Loading Model Organisms We'll load pre-trained model pairs from HuggingFace. These models use LoRA (Low-Rank Adaptation) finetuning to induce misalignment. Our model (Qwen-14b) has 2 LoRA adapters: one which has very high rank and induces strong EM, another whic

related reading