flâneur

Statistics for AI/ML, Part 4: pass@k and Unbiased Estimator

leehanchung.github.io · 1,777 words · saved by 1 readers

A deep dive into the pass@k metric used in LLM benchmarks, explaining the mathematics behind unbiased estimators and why pass@1 doesn't simply mean passing o...

Every time AI labs release new models, we see an evaluation metric called $\text{pass@}k$, where $k$ can be any integer number such as $\text{pass@}1$. It might sound like passing a test at the $k$th attempt, but this metric is far more sophisticated and plays a crucial role in how we build reliable AI applications in production. As an example, here’s OpenAI GPT-5’s performance on AIME. $\text{pass@k}$ does not mean the model passing a test in $k$ attempts. It is calculated using an estimator. Most terms in AI, Machine Learning, and Reinforcement Learning have specific technical definitions th

saved by

related reading