CoT monitorability: why g-means and not F1?

It all has to do with separating monitorability from misalignment property.

Figure 3 in “Monitoring Monitorability” (see below) argues why g-mean is better than F1. I originally didn’t get the implication (esp. about Case B figure) until I understood the mathematical differences between them, and what the implications are when it comes to monitoring misbehaving models. You have a binary monitor that says “misbehavior detected” or “no misbehavior.” The ground truth is either positive (bad behavior happened) or negative (it didn’t). The four outcomes are TP, FP, TN, FN as usual and there are 𝒩 samples, then we get the following operations: TPR = TP/(TP+FN) — “of…

saved by

• Yixiong Hao

related reading

• [2512.18311] Monitoring Monitorabilityarxiv.org
• Should We Train Against (CoT) Monitors? — LessWronglesswrong.com
• Open Sourcing Monitorability Evaluationsalignment.openai.com
• The fragile foundations of CoT monitoring | Christopher Pottsweb.stanford.edu
• A statistical approach to model evaluations \ Anthropicanthropic.com
• Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationarxiv.org
• Logit ROCs: Monitor TPR is linear in FPR in logit spacesubstack.com
• Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationarxiv.org
• Precision and recallen.wikipedia.org
• When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorsarxiv.org
• Early work on monitorability evaluations - METRmetr.org
• Metagaming matters for training, evaluation, and oversightalignment.openai.com