Solvers – Inspect

inspect.ai-safety-institute.org.uk · 3,197 words · saved by 1 readers

Typically, a call to generate() is included in the list of solvers (this solver is just a simple call to the model). You can also create a more sophisticated solver that calls generate() internally, perhaps even more than once (this is often required for more complex evaluations). Next, we’ll describe how solvers operate on task states to do their work. The concept of using solvers and task states for evals was originally introduced in OpenAI Evals. Inspect solvers are an evolution of this core design. Before we get into the specifics of how solvers work, we should describe TaskState, which is the fundamental data structure they act upon. A TaskState consists principally of chat history (derived from input and then extended by model interactions) and model output: Note that the TaskState definition above is simplified: there are other fields in a TaskState but we’re excluding them here for clarity. A prompt engineering solver will modify the content of messages. A model generation solv

Solvers – Inspect Overview Solvers are the heart of Inspect evaluations and can serve a wide variety of purposes, including: Providing system prompts Prompt engineering (e.g. chain of thought) Model generation Self critique Multi-turn dialog Running an agent scaffold Tasks have a single top-level solver that defines an execution plan. This solver could be implemented with arbitrary Python code (calling the model as required) or could consist of a set of other solvers composed together. Solvers can therefore play two different roles: Composite specifications for task execution; and Components t

Solvers – Inspect

related reading