Benchmarking Language Models using the Together Research Computer — TOGETHER

Stanford Center for Research on Foundation Models (CRFM) announced Holistic Evaluation of Language Models (HELM), a comprehensive effort to benchmark 30 language models, including models with limited API access such as OpenAI’s GPT-3 as well as the new emerging ecosystem of open models (e.g., Meta’s

Together’s software network harnessed spare GPU cycles across thousands of servers to benchmark 10 prominent open language models and process 11 billion tokens. We have entered the era of foundation models — massive models trained on huge amounts of data — which can be adapted to a wide range of applications. Language models such as GPT-3 in particular have rich capabilities. They can improve the quality of existing applications (e.g., question answering) and introduce novel applications (e.g., brainstorming slogans or writing blog posts). The pace of innovation is rapid, with new models being

related reading

• Stanford CRFMcrfm.stanford.edu
• Successful language model evals - Jason Weijasonwei.net
• PostTrainBenchposttrainbench.com
• On the Opportunities and Risks of Foundation Modelsarxiv.org
• KernelBench: Can LLMs Write GPU Kernels?scalingintelligence.stanford.edu
• Tutorial - CRFM HELMcrfm-helm.readthedocs.io
• GitHub - open-compass/opencompass: OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.github.com
• GLM-5.2: Built for Long-Horizon Tasksz.ai
• More Efficient In-Context Learning with GLaMblog.research.google
• Language Model Compositionarxiv.org
• Language Models can Solve Computer Tasksarxiv.org
• BigO(Bench) -- Can LLMs Generate Code with Controlled Time and Space Complexity?alphaxiv.org