✳flâneur — a map of the web's best reading

Extrapolating GPT-N performance

alignmentforum.org · 20,732 words · saved by 1 readers

The GPT-3 paper (Brown et al., 2020) contains measurements of how 8 transformers of different sizes perform on lots of different benchmarks. In this post, I extrapolate them forward and give an overview of issues that could appear when scaling up. Note that these benchmarks are for ‘downstream tasks’ that are different from the training task (which is to predict the next token); these extrapolations thus cannot be directly read off the scaling laws in OpenAI’s Scaling Laws for Neural Language Models (Kaplan et al., 2020) or Scaling Laws for Autoregressive Generative Modelling (Henighan et al., 2020). (If you don’t care about methodology or explanations, the final graphs are in Comparisons and limits .) METHODOLOGY Brown et al. reports benchmark performance for 8 different model sizes. However, these models were not trained in a compute-optimal fashion. Instead, all models were trained on 300B tokens (one word is roughly 1.4 tokens), which is inefficiently much data. Since we’re interested in the best performance we can get for a given amount of compute, and these models weren’t compute-optimally trained, we cannot extrapolate these results on the basis of model-size. Instead, I fit a trend for how benchmark performance (measured in % accuracy) depends on the cross-entropy loss that the models get when predicting the next token on the validation set. I then use the scaling laws from Scaling Laws for Neural Language Models to extrapolate this loss. This is explained in the Appendix. PLOTTING AGAINST LOSS In order to get a sense of how GPT-3 performs on different types of tasks, I separately report few-shot progress on each of the 11 different categories discussed in Brown et al. For a fair comparison, I normalize the accuracy of each category between random performance and maximum performance; i.e., for each data point, I subtract the performance that a model would get if it responded randomly (or only responded with the most common answer), and divide by the dif

x Extrapolating GPT-N performance — AI Alignment Forum Extrapolating GPT-N performance AI Timelines GPT Language Models (LLMs) AI Frontpage 39 Extrapolating GPT-N performance by Lukas Finnveden 18th Dec 2020 27 min read 31 39 Brown et al. (2020) (which describes the development of GPT-3) contains measurements of how 8 transformers of different sizes perform on several different benchmarks. In this post, I project how performance could improve for larger models, and give an overview of issues that may appear when scaling-up. Note that these benchmarks are for ‘downstream tasks’ that are differe

Explore this link on the map →

related reading