Fine-Tuned Lie Detectors Failed to Generalize
We trained lie detectors on on-policy lies from open-source models, but fine-tuning them didn’t generalize well to out-of-distribution cases. In these cases, fine-tuned detectors barely beat prompted baselines, and larger prompted models often beat them outright. Larger models were generally better at detecting lies, though the trend was not monotonic. To support further research, we publicly release our datasets here. Research done as part of MATS and the Anthropic Fellowship. Misalignment is hardest to correct when it's concealed: a model that pursues the wrong goals or holds false beliefs in the open gives operators something to act on. This is why a reliable lie detector would be useful to many oversight schemes. It would not make a model aligned, but it raises the cost of hiding. That said, a lie detector only scores assertions, and a model can conceal a great deal without asserting anything false; and detection also only helps if operators have a channel to correct what it finds.