EleutherAI/lm-evaluation-harness: A framework for few-shot evaluation of autoregressive language models.

A framework for few-shot evaluation of autoregressive language models.

Language Model Evaluation Harness Latest News 📣 [2025/12] CLI refactored with subcommands ( run , ls , validate ) and YAML config file support via --config . See the CLI Reference and Configuration Guide . [2025/12] Lighter install : Base package no longer includes transformers / torch . Install model backends separately: pip install lm_eval[hf] , lm_eval[vllm] , etc. [2025/07] Added think_end_token arg to hf (token/str), vllm and sglang (str) for stripping CoT reasoning traces from models that support it. [2025/03] Added support for steering HF models! [2025/02] Added SGLang support! [2024/0

related reading

• Patterns for Building LLM-based Systems & Productseugeneyan.com
• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• GitHub - open-compass/opencompass: OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.github.com
• GitHub - Hannibal046/Awesome-LLM: Awesome-LLM: a curated list of Large Language Modelgithub.com
• Llama 2 · Hugging Facehuggingface.co
• LLM Evaluation doesn't need to be complicatedphilschmid.de
• GitHub - NVIDIA-NeMo/Automodel: 🚀 Pytorch Distributed native training library for LLMs/VLMs with OOTB Hugging Face supportgithub.com
• LLM evaluation: a beginner's guideevidentlyai.com
• Your AI Product Needs Evals – Hamel's Blog - Hamel Husainhamel.dev
• Successful language model evals - Jason Weijasonwei.net
• GitHub - rasbt/LLMs-from-scratch: Implement a ChatGPT-like LLM in PyTorch from scratch, step by stepgithub.com
• GitHub - guidance-ai/guidance: A guidance language for controlling large language models.github.com