2.4: Scaling Laws | AI Safety, Ethics, and Society Textbook

Compelling evidence shows that increases in the performance of many AI systems can be modeled with equations called scaling laws. Common knowledge suggests that larger models with more data will perform better, frequently reiterated in phrases like “add more layers” or “use more data.” Scaling laws make this folk knowledge mathematically precise. In this section, we show that the performance of a deep learning model scales according to parameter count and dataset size—both of which are primarily bottlenecked by the computational resources available. Scaling laws describe the relationship between a model’s performance and primary inputs. Power laws are mathematical equations that model how a particular quantity varies as the power of another. In power laws, the variation in one quantity is proportional to a power (exponent) of the variation in another. The power law y = bxa states that the change in y is directly proportional to the change in x raised to a certain power a. If a is 2, th

Compelling evidence shows that increases in the performance of many AI systems can be modeled with equations called scaling laws. Common knowledge suggests that larger models with more data will perform better, frequently reiterated in phrases like “add more layers” or “use more data.” Scaling laws make this folk knowledge mathematically precise. In this section, we show that the performance of a deep learning model scales according to parameter count and dataset size—both of which are primarily bottlenecked by the computational resources available. Scaling laws describe the relationship betwe