My benchmark for large language models

nicholas.carlini.com · 2,315 words · saved by 1 readers

I've just released a new benchmark for large language models on my GitHub. It's a collection of nearly 100 tests I've extracted from my actual conversation history with various LLMs. Among the tests included in the benchmark are tests that ask a model to There are two defining features of this benchmark that make it interesting. Most importantly, I've implemented a simple dataflow domain specific language to make it easy for me (or anyone else!) to add new tests that realistically evaluate model capabilities. This DSL allows for specifying both how the question should be asked and also how the answer should be evaluated. Most questions are evalauted by actually running the code the model writes [a] but the framework supports a bunch of other evaluation methods as well. And then, directly as a result of this, I've written nearly 100 tests for different situations I've actually encountered when working with LLMs as assistants. [b] For example, here's the test case that evaluates if a mod

My benchmark for large language models Main Papers Talks Code Writing Writing My benchmark for large language models by Nicholas Carlini 2024-02-19 I've just released a new benchmark for large language models on my GitHub . It's a collection of nearly 100 tests I've extracted from my actual conversation history with various LLMs. Among the tests included in the benchmark are tests that ask a model to convert a python function to an equivalent-but-faster c function; explain the functionality of minified javascript; identify the encoding format (in this case, uuencoded) of some data; write a par

My benchmark for large language models

related reading