Metrics — Continual Learning Bench Docs

Each task is designed to answer one question: does the system perform better the more experience it has? Each task defines a per-instance reward. Tasks have their own definition (accuracy, normalized error, profit, …), but every reward must satisfy: If per-instance difficulty is roughly homogeneous across the sequence, reward by itself is enough to read off learning: a system that is genuinely improving from experience will produce an upward-trending reward curve, because the curve isn't being moved by changes in instance difficulty. In that regime, "higher reward at task instance t" means "more was learned by instance t." For example, in the "exploitable poker" task, chips are reset after each hand (so winnings are not accumulated). That means that when a system is playing against the same opponent, the basic challenge for that hand is the same as in previous hands. Increases in reward across hands can thus reflect real learning signal — systems are doing progressively better against

Each task is designed to answer one question: does the system perform better the more experience it has? Reward Each task defines a per-instance reward. Tasks have their own definition (accuracy, normalized error, profit, …), but every reward must satisfy: Higher is better. Comparable across different instances of a task. Reflects the task objective directly, not an unrelated proxy. If per-instance difficulty is roughly homogeneous across the sequence, reward by itself is enough to read off learning: a system that is genuinely improving from experience will produce an upward-trending…

saved by

• Maanav Khaitan

related reading

• Concepts — Continual Learning Bench Docscontinual-learning-bench.com
• Continual Learning Bench 1.0 — Newscontinual-learning-bench.com
• Neural Cheat Sheets: Learning to Summarize with Reinforcement Learning | Applied Computeappliedcompute.com
• Reward is not the optimization target — LessWronglesswrong.com
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• You can’t imitation-learn how to continual-learn — LessWronglesswrong.com
• The Continual Learning Problemjessylin.com
• Models Don't "Get Reward" — LessWronglesswrong.com
• Reinforcement Learning With Verifiable Rewards: How Data and Verifiers Shape RLVRsnorkel.ai
• Reward Is Not the Optimization Targetturntrout.com
• IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io