What's going on with the Open LLM Leaderboard?

The discussion centered around one of the four evaluations displayed on the leaderboard: a benchmark for measuring Massive Multitask Language Understanding (shortname: MMLU). The community was surprised that MMLU evaluation numbers of the current top model on the leaderboard, the LLaMA model 🦙, were significantly lower than the numbers in the published LLaMa paper. So we decided to dive in a rabbit hole to understand what was going on and how to fix it 🕳🐇 In our quest, we discussed with both the great @javier-m who collaborated on the evaluations of LLaMA and the amazing @slippylolo from the Falcon team. This being said, all the errors in the below should be attributed to us rather than them of course! Along this journey with us you’ll learn a lot about the ways you can evaluate a model on a single evaluation and whether or not to believe the numbers you see online and in papers. Ready? Then buckle up, we’re taking off 🚀. First, note that the Open LLM Leaderboard is actually just a

Recently an interesting discussion arose on Twitter following the release of Falcon 🦅 and its addition to the Open LLM Leaderboard, a public leaderboard comparing open access large language models. The discussion centered around one of the four evaluations displayed on the leaderboard: a benchmark for measuring Massive Multitask Language Understanding (shortname: MMLU). The community was surprised that MMLU evaluation numbers of the current top model on the leaderboard, the LLaMA model 🦙, were significantly lower than the numbers in the published LLaMa paper. So we decided to dive in a…

related reading

• Successful language model evals - Jason Weijasonwei.net
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• Large Language Model: world models or surface statistics?thegradient.pub
• LLM Evaluation doesn't need to be complicatedphilschmid.de
• LLM evaluation: a beginner's guideevidentlyai.com
• GitHub - Hannibal046/Awesome-LLM: Awesome-LLM: a curated list of Large Language Modelgithub.com
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• GitHub - open-compass/opencompass: OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.github.com
• The Llama Hitchiking Guide to Local LLMs – hackerllamaosanseviero.github.io
• Things we learned about LLMs in 2024simonwillison.net