Qian XIAO (@qianxiao): "Zilan Qian’s analysis of different understandings of loss of control (LoC) is particularly interesting. It reminded me of one of our dialogues with U.S. colleagues, where we spent considerable time discussing what we actually meant by “loss of control.” What initially appeared …"
Zilan Qian’s analysis of different understandings of loss of control (LoC) is particularly interesting. It reminded me of one of our dialogues with U.S. colleagues, where we spent considerable time discussing what we actually meant by “loss of control.” What initially appeared to be a shared term turned out to contain rather different assumptions about the nature and sources of AI risk. The two sides appeared to differ somewhat in how they conceptualized LoC. Chinese experts placed greater emphasis on operational control structures, governance mechanisms, and system reliability: how deployed AI systems can remain under effective human supervision, and how misuse, manipulation, unintended behavior, and system failures can be prevented or corrected. U.S. experts, by contrast, tended to focus more on capability thresholds and longer-term risks, often invoking concepts such as artificial general intelligence (AGI), artificial superintelligence (ASI), and the technological singularity to describe potential future inflection points. In this framing, loss of control can be understood as a discrete future threshold: once AI systems approach or surpass human capabilities—or acquire the ability to improve themselves autonomously—the nature and scale of the risks could change fundamentally. From the perspective of some Chinese experts in our discussions, however, AI loss of control is better understood not as a single catastrophic event or a discrete future threshold, but as a progressively emerging engineering failure mode. Rather than occurring suddenly at a particular capability threshold, loss of control may develop incrementally as AI systems become more complex and autonomous and increasingly difficult for humans to understand, supervise, and constrain. Chinese experts highlighted three features in particular. First, the complexity of AI systems may gradually exceed the capacity for effective human oversight. Second, verification mechanisms over extended time horizons