flâneur

Ankur Goyal on X: "Evals ≠ tests" / X

x.com · 807 words · saved by 1 readers

Evals ≠ tests

While building Braintrust I’ve been thinking about what evals are good at, what they should be used for, and how they differ from other parts of the software development lifecycle, like tests. Though it’s tempting to conflate evals and tests, doing so confuses the differences between them. Here’s how we think about it at Braintrust: A test is about if the thing can work An eval is about what the thing can and cannot do A mature eval practice is about what the thing should be doing, how often, and in what circumstances The last one is important when building agents. You need to test…

saved by

related reading