flâneur

Highly optimized optimizers - by Ben Recht - arg min

argmin.net · 951 words · saved by 1 readers

This is a live blog of Lecture 10 of the 2025 edition of my graduate machine learning class “Patterns, Predictions, and Actions.” A Table of Contents is here. It’s undeniable that everything in machine learning is an optimization problem. The fundamental problem of machine learning is an optimization problem: minimizing average prediction errors on data we haven’t seen yet. In practice, we more or less do this by minimizing average prediction errors on the data we’ve collected so far. Given its fundamental position, how much numerical optimization should we learn in a machine learning class? I’m not sure I like my answer, but I’ve converged on a single week dive into stochastic gradient methods. The vast majority of machine learning problems are now solved using some variant of such methods. And they are the common core connecting the first machine algorithm—the perceptron—to modern algorithms like the default-solver-of-all-problems ADAM and advanced reinforcement learning algorithms l

Highly optimized optimizers Justifying a laser focus on stochastic gradient methods. Ben Recht Oct 02, 2025 22 13 Share This is a live blog of Lecture 10 of the 2025 edition of my graduate machine learning class “Patterns, Predictions, and Actions.” A Table of Contents is here . It’s undeniable that everything in machine learning is an optimization problem. The fundamental problem of machine learning is an optimization problem: minimizing average prediction errors on data we haven’t seen yet. In practice, we more or less do this by minimizing average prediction errors on the data we’ve collect

saved by

related reading