flâneur

Reasoning models don't always say what they think \ Anthropic

anthropic.com · 1,889 words · saved by 1 readers

Since late last year, “reasoning models” have been everywhere. These are AI models—such as Claude 3.7 Sonnet—that show their working: as well as their eventual answer, you can read the (often fascinating and convoluted) way that they got there, in what’s called their “Chain-of-Thought”. As well as helping reasoning models work their way through more difficult problems, the Chain-of-Thought has been a boon for AI safety researchers. That’s because we can (among other things) check for things the model says in its Chain-of-Thought that go unsaid in its output, which can help us spot undesirable behaviours like deception. But if we want to use the Chain-of-Thought for alignment purposes, there’s a crucial question: can we actually trust what models say in their Chain-of-Thought? In a perfect world, everything in the Chain-of-Thought would be both understandable to the reader, and it would be faithful—it would be a true description of exactly what the model was thinking as it reached its a

Alignment Reasoning models don't always say what they think Apr 3, 2025 Read the paper Since late last year, “reasoning models” have been everywhere. These are AI models—such as Claude 3.7 Sonnet—that show their working : as well as their eventual answer, you can read the (often fascinating and convoluted) way that they got there, in what’s called their “Chain-of-Thought”. As well as helping reasoning models work their way through more difficult problems, the Chain-of-Thought has been a boon for AI safety researchers. That’s because we can (among other things) check for things the model says i

saved by

related reading