✳flâneur — a map of the web's best reading

Online KL Shampoo | Tilde

blog.tilderesearch.com · 13,632 words · saved by 1 readers

Modern neural-network optimizers can largely be understood as different ways of preconditioning the gradient. Given a gradient 𝑔 g, an optimizer applies a transformation that changes the scale and orientation of the resulting update. At one extreme, stochastic gradient descent applies no adaptive preconditioning, while Adam maintains a diagonal estimate of the gradient second moment. Full-matrix AdaGrad [3] instead maintains and updates using This full-matrix update is attractive because it accounts for correlations among parameters rather than treating each coordinate independently. By equalizing the magnitude of updates across different directions, it achieves optimal regret bounds in adversarial online learning environments. Unfortunately, the full AdaGrad matrix grows quadratically with the number of parameters. It is intractable for modern neural networks. Shampoo [4] makes full-matrix AdaGrad more practical by approximating its preconditioner with a Kronecker product. For a mat

Back Online KL Shampoo 7.28.2026 Ashley Zhang*, Ben Keigwin*, Dhruv Pai*, Alec Dewulf * Core Contributor; Correspondence to ashleyzhang@tilderesearch.com cite ↓ TL;DR We introduce Online KL Shampoo (OKLS), a zero-staleness, hardware-aligned, Kronecker product-based optimizer that achieves 1.59× Muon's parameter efficiency at the 1.2B Muon-equivalent scale , with the advantage increasing toward larger models, while maintaining 98% training throughput. We developed Scaled CANS Coupled Newton-Schulz , an iterative root-finding method that computes the − 1 / 2 -1/2 − 1/2 matrix root using 27 FP16

Explore this link on the map →

saved by

related reading