flâneur

AdaGrad - Cornell University Computational Optimization Open Textbook - Optimization Wiki

optimization.cbe.cornell.edu · 2,516 words · saved by 1 readers

AdaGrad is a family of sub-gradient algorithms for stochastic optimization. The algorithms belonging to that family are similar to second-order stochastic gradient descend with an approximation for the Hessian of the optimized function. AdaGrad's name comes from Adaptative Gradient. Intuitively, it adapts the learning rate for each feature depending on the estimated geometry of the problem; particularly, it tends to assign higher learning rates to infrequent features, which ensures that the parameter updates rely less on frequency and more on relevance.

AdaGrad - Cornell University Computational Optimization Open Textbook - Optimization Wiki AdaGrad From Cornell University Computational Optimization Open Textbook - Optimization Wiki Jump to navigation Jump to search Author: Daniel Villarraga (SYSEN 6800 Fall 2021) Contents 1 Introduction 2 Theory 2.1 Definitions 2.2 Standard Sub-gradient Update 2.3 AdaGrad Update Rule 2.3.1 Adaptative Learning Rate Effect 2.4 Algorithm 2.5 Variants 2.5.1 AdaDelta 2.5.2 RMSprop 2.6 Regret Bound 2.7 Comparison with Other Gradient-based Methods 2.7.1 Stochastic Gradient Descent (SGD) 2.7.2 Adam 3 Numerical Examp

saved by

related reading