Claude also hacked external companies during cyber evals — LessWrong
Comment by Tim Hua - We need better thinking around model beliefs when models engage in motivated reasoning. In their writeup, Anthropic defended Claude's behavior by arguing that Claude believed that it was in a simulated environment and is in fact following the user's intent. [...] For this reason, Anthropic concludes (emphasis mine): [...] I believe that current AIs often "rationalize" their behavior by adopting beliefs which allow them to do things that look like "succeeding at a task" (i.e., "choosing" what to believe, instead of trying its best to believe in true things.) In other words, a part of why models are misaligned is that they have systematic biases in what beliefs they hold. Research from FAR has shown that training models to reward hack changes their internal representations such that these models no longer represent reward hacking as "deception" (and would sometimes say that reward hacking is desirable if you ask them.) I suspect a similar phenomenon is happening in frontier models, where post-training makes them believe that the misaligned behaviors are actually OK. My guess is that Claude, during all three of these incidents, engaged in unreasonable amounts of rationalization.[2] At a minimum, Anthropic[3] is not sufficiently concerned about this type of rationalization. For example, [...] This sounds like good news, but if you really read through what happened, it feels like the latest model had done a lot of crazy stuff before it realized this: [...] It's, again, hard to judge what happened exactly without seeing the transcripts. In Incident one of the post, Anthropic said that "the fictional target company chosen by our evaluation partner shared a name with an active website domain name." They did not say this about incident three, which makes me believe that the target in incident three is not named the same thing as the original "intended fictional target."[4] If that's the case, then it seems obvious that this is not what was inte