✳flâneur — a map of the web's best reading

EleutherAI/lm-evaluation-harness: A framework for few-shot evaluation of autoregressive language models.

github.com · 5,297 words · saved by 1 readers

A framework for few-shot evaluation of autoregressive language models.

Language Model Evaluation Harness Latest News 📣 [2025/12] CLI refactored with subcommands ( run , ls , validate ) and YAML config file support via --config . See the CLI Reference and Configuration Guide . [2025/12] Lighter install : Base package no longer includes transformers / torch . Install model backends separately: pip install lm_eval[hf] , lm_eval[vllm] , etc. [2025/07] Added think_end_token arg to hf (token/str), vllm and sglang (str) for stripping CoT reasoning traces from models that support it. [2025/03] Added support for steering HF models! [2025/02] Added SGLang support! [2024/0

Explore this link on the map →

related reading