flâneur

RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing | LMSYS Org

lmsys.org · 1,499 words · saved by 1 readers

by: Isaac Ong*, Amjad Almahairi*, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, Ion Stoica, Jul 01, 2024 LLMs have demonstrated remarkable capabilities across a range of tasks, but there exists wide variation in their costs and capabilities, as seen from the plot of performance against cost in Figure 1. Very broadly, more capable models tend to be more expensive than less capable models. This leads to a dilemma when deploying LLMs in the real-world - routing all queries to the largest, most capable model leads to the highest-quality responses but can be expensive, while routing queries to smaller models can save costs but may result in lower-quality responses. Figure 1: Plot of performance against cost of various LLMs. Performance is measured by Elo on Chatbot Arena, and cost per million tokens assuming a 1:1 input / output ratio. Through routing between two models, we ideally achieve a better performance:cost ratio than can be achieved with either model.

RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing - LMSYS Org Projects Blog About Donations Contact ‹ Back to Blog ‹ Back to Blog Contents Routing Setup Results RouteLLM vs Commercial Offerings Generalizing to Other Models Conclusion Acknowledgements Citations RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing Isaac Ong*, Amjad Almahairi*, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, Ion Stoica July 1, 2024 LLMs have demonstrated remarkable capabilities across a range of tasks, but there exists wide variation in their costs and cap

related reading