The cost of becoming good - by Lydia Nottingham - pronotre

lydianottingham.substack.com · 414 words · saved by 1 readers

This is part one of a two-part series on the lure of amateurism. Part Two will be about why people avoid specialization.¹ if you usually lurk, consider subscribing; i email <2x/year While working on the SvR gap, I ran into a simple unpleasant observation: If SvR coherence is actually a dangerous capability—i.e. a move that increases loss-of-control risk—then I should be cautious about pushing it. Right now, I get to be curious because I’m small. My experiments are micro-scale, within the ambient research distribution. Nothing I do is out-of-distribution enough to bear real moral load. But if that changed—if I started doing more robust or decorrelated things—I’d inherit obligations I currently avoid. And I can feel the incentive to not let those conditions change, because I like the freedom I have. That’s the pull of amateurism.² Competence tightens the constraint set. Becoming actually good implies a heavier burden of evaluation: what I do, why I do it, which trajectory i’m nudging. Th

Part One: 'The cost of mastery’ || Part Two: ‘The cost of specialization’ While working on research into increasing LM consistency, I ran into an unpleasant observation: If consistency is actually a dangerous capability, then I should be cautious about pushing it. Right now, I get to be curious because I’m small. My experiments are micro-scale, within the ambient research distribution. Nothing I do is out-of-distribution enough to bear real moral load. But if that changed—if I started doing more robust or decorrelated things—I’d inherit obligations I currently avoid. And I can feel the…

The cost of becoming good - by Lydia Nottingham - pronotre

related reading