Contra Dwarkesh on RL sample-efficiency via information theory

Dwarkesh Patel wrote an article RL is even more information inefficient than you thought. I’ve been trying to understand RL recently so I read the post with a lot of interest; but I think the main technical point in the post is wrong. Without commenting on the broader point about RL sample-efficiency in general, in this post I claim that information-theoretic entropy of labels is not the right way to think about learning. Dwarkesh’s post tries to compare “the amount of new information you can extract” in reinforcement learning vs supervised learning. For simplicity, he assumes the model is predicting a single token. The two settings are: Supervised learning: we update on the correct token RL: the model predicts a token, and gets a binary outcome (correct or not). Dwarkesh’s claim is that the information gained in these two settings depends strongly on the pass rate p, which is the probability of getting the correct token. Concretely, Dwarkesh computes the information gain as -log(p) fo

Dwarkesh Patel wrote an article RL is even more information inefficient than you thought. I’ve been trying to understand RL recently so I read the post with a lot of interest; but I think the main technical point in the post is wrong. Without commenting on the broader point about RL sample-efficiency in general, in this post I claim that information-theoretic entropy of labels is not the right way to think about learning. Dwarkesh’s post tries to compare “the amount of new information you can extract” in reinforcement learning vs supervised learning. For simplicity, he assumes the model is pre