SGD — PyTorch 2.4 documentation

Nesterov momentum is based on the formula from On the importance of initialization and momentum in deep learning. params (iterable) – iterable of parameters to optimize or dicts defining parameter groups lr (float, optional) – learning rate (default: 1e-3) momentum (float, optional) – momentum factor (default: 0) weight_decay (float, optional) – weight decay (L2 penalty) (default: 0) dampening (float, optional) – dampening for momentum (default: 0) nesterov (bool, optional) – enables Nesterov momentum (default: False) maximize (bool, optional) – maximize the objective with respect to the params, instead of minimizing (default: False) foreach (bool, optional) – whether foreach implementation of optimizer is used. If unspecified by the user (so foreach is None), we will try to use foreach over the for-loop implementation on CUDA, since it is usually significantly more performant. Note that the foreach implementation uses ~ sizeof(params) more peak memory than the for-loop version due to

Redirecting… Continue to ../../2.13/generated/torch.optim.SGD.html

related reading

• Redirecting…pytorch.org
• Stochastic gradient descent - Wikipediaen.m.wikipedia.org
• An overview of gradient descent optimization algorithmsruder.io
• Deriving Muonjeremybernste.in
• Learning PyTorch with Examples — PyTorch Tutorials 2.12.0+cu130 documentationpytorch.org
• Redirecting…pytorch.org
• Why Momentum Really Worksdistill.pub
• [1512.04202] Preconditioned Stochastic Gradient Descentarxiv.org
• microgptkarpathy.github.io
• [2411.19870] DeMo: Decoupled Momentum Optimizationarxiv.org
• AdaGrad - Cornell University Computational Optimization Open Textbook - Optimization Wikioptimization.cbe.cornell.edu
• PyTorch internals : ezyang's blogblog.ezyang.com