Prompt-to-Leaderboard

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions.

Prompt-to-Leaderboard Evan Frick ∗ , Connor Chen ∗ , Joseph Tennyson ∗ , Tianle Li ∗ , Wei-Lin Chiang ∗ , Anastasios N. Angelopoulos ∗ , Ion Stoica {evanfrick, connorchen, josephtennyson, tianleli, weichiang, angelopoulos, istoica}@berkeley.edu (University of California, Berkeley March 10, 2025 *equal contribution ) Abstract Large language model (LLM) evaluations typically rely on aggregated metrics like accuracy or human preference, averaging across users and prompts. This averaging obscures user- and prompt-specific variations in model performance. To address this, we propose Prompt-to-Leade

related reading

• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Surfacing Pathological Behaviors in Language Models | Transluce AItransluce.org
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• [2507.19457] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• 2401.10020.pdfarxiv.org
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelsarxiv.org
• Mechanistically Eliciting Latent Behaviors in Language Models — AI Alignment Forumalignmentforum.org
• 2402.07927arxiv.org
• RLHF | John Lambertjohnwlambert.github.io
• Language Models can Solve Computer Tasksarxiv.org