flâneur

AI safety techniques leveraging distillation — LessWrong

lesswrong.com · 4,018 words · saved by 1 readers

It's currently possible to (mostly or fully) cheaply reproduce the performance of a model by training another (initially weaker) model to imitate the stronger model's outputs.[1] I'll refer to this as distillation. In the case of RL, distilling the learned capabilities is much, much cheaper than the RL itself (especially if you are distilling back into the original base model). But even for pre-training, distilling is cheaper than the original training.[2] In this post, I'll discuss how we could utilize distillation to potentially remove (or possibly detect) misalignment. I'll also discuss a few other applications.[3] My overall take is that techniques utilizing distillation are mildly to moderately promising and the low cost of distillation might make them surprisingly viable, but it's quite tricky to reason about how effective these techniques are. I'll assume that we have a powerful model[4] that we're worried is seriously misaligned and we wish to use distillation to mitigate this.

x AI safety techniques leveraging distillation — LessWrong AI Frontpage 62 AI safety techniques leveraging distillation by ryan_greenblatt 19th Jun 2025 AI Alignment Forum 14 min read 0 62 Ω 36 It's currently possible to (mostly or fully) cheaply reproduce the performance of a model by training another (initially weaker) model to imitate the stronger model's outputs. [1] I'll refer to this as distillation. In the case of RL, distilling the learned capabilities is much, much cheaper than the RL itself (especially if you are distilling back into the original base model). But even for pre-trainin

related reading