flâneur

Concepts — Continual Learning Bench Docs

continual-learning-bench.com · 637 words · saved by 1 readers

A system is what we aim to evaluate. It is the combination of an agent that responds to queries from a task and a memory system that carries information between task instances. The memory system might be pure in-context learning (perhaps with caching), lossy context compression (e.g. periodic summarization), parametric (fine-tuning, weight updates), or any combination of these. The benchmark scores neither component in isolation: performance on continual learning is a property of the two together. So we can ask questions like how well an agent does on tasks for a given memory system and how effective memory systems are across agents and tasks. Systems live under src/systems/ and implement the ContinualLearningSystem interface. A task is a continual learning problem. It has a theme (poker, codebase, etc) and consists of a sequence of task instances that systems attempt to solve. Every task lives under src/tasks/ and implements the ContinualLearningTask interface. Different tasks are unr

System A system is what we aim to evaluate. It is the combination of an agent that responds to queries from a task and a memory system that carries information between task instances. The memory system might be pure in-context learning (perhaps with caching), lossy context compression (e.g. periodic summarization), parametric (fine-tuning, weight updates), or any combination of these. The benchmark scores neither component in isolation: performance on continual learning is a property of the two together. So we can ask questions like how well an agent does on tasks for a given memory system…

saved by

related reading