✳flâneur — a map of the web's best reading

Tutorial - CRFM HELM

crfm-helm.readthedocs.io · 694 words · saved by 1 readers

This tutorial will explain how to use the HELM command line tools to run benchmarks, aggregate statistics, and visualize results. We will run two runs using the mmlu scenario on the huggingface/gpt-2 model. The mmlu scenario implements the Massive Multitask Language (MMLU) benchmark from this paper, and consists of a Question Answering (QA) task using a dataset with questions from 57 subjects such as elementary mathematics, US history, computer science, law, and more. Note that GPT-2 performs poorly on MMLU, so this is just a proof of concept. We will run two runs: the first using questions about anatomy, and the second using questions about philosophy. helm-run is a command line tool for running benchmarks. To run this benchmark using the HELM command-line tools, we need to specify run spec descriptions that describes the desired runs. For this example, the run spec descriptions are mmlu:subject=anatomy,model=huggingface/gpt-2 (for anatomy) and mmlu:subject=philosophy,model=huggingfac

Tutorial - CRFM HELM User Guide Tutorial Edit on GitHub Tutorial This tutorial will explain how to use the HELM command line tools to run benchmarks, aggregate statistics, and visualize results. We will run two runs using the mmlu scenario on the openai/gpt2 model. The mmlu scenario implements the Massive Multitask Language (MMLU) benchmark from this paper , and consists of a Question Answering (QA) task using a dataset with questions from 57 subjects such as elementary mathematics, US history, computer science, law, and more. Note that GPT-2 performs poorly on MMLU, so this is just a proof of

Explore this link on the map →

related reading