[1805.00899] AI safety via debate

ar5iv.labs.arxiv.org · 14,490 words · saved by 1 readers

To make AI systems broadly useful for challenging real-world tasks, we need them to learn complex human goals and preferences. One approach to specifying complex goals asks humans to judge during training which agent behaviors are safe and useful, but this approach can fail if the task is too complicated for a human to directly judge. To help address this concern, we propose training agents via self play on a zero sum debate game. Given a question or proposed action, two agents take turns making short statements up to a limit, then a human judges which of the agents gave the most true, useful information. In an analogy to complexity theory, debate with optimal play can answer any question in \PSPACE given polynomial time judges (direct judging answers only \NP questions). In practice, whether debate works involves empirical questions about humans and the tasks we want AIs to perform, plus theoretical questions about the meaning of AI alignment. We report results on an initial MNIST

\newclass \DEBATE DEBATE \newclass \QBF QBF AI safety via debate Geoffrey Irving Corresponding author: irving@openai.com Paul Christiano OpenAI Dario Amodei Abstract To make AI systems broadly useful for challenging real-world tasks, we need them to learn complex human goals and preferences. One approach to specifying complex goals asks humans to judge during training which agent behaviors are safe and useful, but this approach can fail if the task is too complicated for a human to directly judge. To help address this concern, we propose training agents via self play on a zero sum debate game.

[1805.00899] AI safety via debate

related reading