✳flâneur — a map of the web's best reading

[2512.18311] Monitoring Monitorability

arxiv.org · 751 words · saved by 1 readers

Abstract:Observability into the decision making of modern AI systems may be required to safely deploy increasingly capable agents. Monitoring the chain-of-thought (CoT) of today's reasoning models has proven effective for detecting misbehavior. However, this "monitorability" may be fragile under different training procedures, data sources, or even continued system scaling. To measure and track monitorability, we propose three evaluation archetypes (intervention, process, and outcome-property) and a new monitorability metric, and introduce a broad evaluation suite. We demonstrate that these evaluations can catch simple model organisms trained to have obfuscated CoTs, and that CoT monitoring is more effective than action-only monitoring in practical settings. We compare the monitorability of various frontier models and find that most models are fairly, but not perfectly, monitorable. We also evaluate how monitorability scales with inference-time compute, reinforcement learning optimization, and pre-training model size. We find that longer CoTs are generally more monitorable and that RL optimization does not materially decrease monitorability even at the current frontier scale. Notably, we find that for a model at a low reasoning effort, we could instead deploy a smaller model at a higher reasoning effort (thereby matching capabilities) and obtain a higher monitorability, albeit at a higher overall inference compute cost. We further investigate agent-monitor scaling trends and find that scaling a weak monitor's test-time compute when monitoring a strong agent increases monitorability. Giving the weak monitor access to CoT not only improves monitorability, but it steepens the monitor's test-time compute to monitorability scaling trend. Finally, we show we can improve monitorability by asking models follow-up questions and giving their follow-up CoT to the monitor.

[2512.18311] Monitoring Monitorability Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Artificial Intelligence arXiv:2512.18311 (cs) [Submitted on 20 Dec 2025] Title: Monitoring Monitorability Authors: Melody Y. Guan , Miles Wang , Micah Carroll , Zehao Dou , Annie Y. Wei , Marcus Williams , Benjamin Arnav , Joost Huizinga , Ian Kivlichan , Mia Glaese , Jakub Pachocki , Bowen Baker View a PDF of the paper titled Monitoring Monitorability, by Melody Y. Guan and 11 other authors View

Explore this link on the map →

related reading