RL ALGO

Here's my attempt to answer all the questions as best i could. Happy to be corrected and update my understanding. Q. Why Actor-Critic instead of pure Critic A pure critic (value based like DQN) need an argmax over actions, for LLMs the action space is the entire vocab so argmax is dead on arrival and impossible for continuous control. Actor-critic handles continuous action spaces naturally. Actor-critic has lower variance than pure policy gradient (REINFORCE). pure policy handles big action spaces fine but the updates are high variance since youre using full returns. actor-critic keeps a parameterized policy and uses the critic as baseline to kill that variance, plus the critic lets you bootstrap so credit assignment doesnt have to wait for the whole episode. one thing to note > in LLM RL the actor-critic argument is actually weaker than in classical RL because value function over token sequence is hard to learn well, which is exactly why GRPO throws the critic away and just uses a gro

saved by

• KRUPAL VIRANI