flâneur

SGD — PyTorch 2.4 documentation

pytorch.org · 2,062 words · saved by 1 readers

Nesterov momentum is based on the formula from On the importance of initialization and momentum in deep learning. params (iterable) – iterable of parameters to optimize or dicts defining parameter groups lr (float, optional) – learning rate (default: 1e-3) momentum (float, optional) – momentum factor (default: 0) weight_decay (float, optional) – weight decay (L2 penalty) (default: 0) dampening (float, optional) – dampening for momentum (default: 0) nesterov (bool, optional) – enables Nesterov momentum (default: False) maximize (bool, optional) – maximize the objective with respect to the params, instead of minimizing (default: False) foreach (bool, optional) – whether foreach implementation of optimizer is used. If unspecified by the user (so foreach is None), we will try to use foreach over the for-loop implementation on CUDA, since it is usually significantly more performant. Note that the foreach implementation uses ~ sizeof(params) more peak memory than the for-loop version due to

class torch.optim.SGD(params, lr=0.001, momentum=0, dampening=0, weight_decay=0, nesterov=False, *, maximize=False, foreach=None, differentiable=False, fused=None)[source]# Implements stochastic gradient descent (optionally with momentum). input:γ (lr), θ0 (params), f(θ) (objective), λ (weight decay), μ (momentum), τ (dampening), nesterov, maximizefor t=1 to … doif maximize:gt←−∇θft(θt−1)elsegt←∇θft(θt−1)if λ≠0gt←gt+λθt−1if μ≠0if t>1bt←μbt−1+(1−τ)gtelsebt←gtif nesterovgt←gt+μbtelsegt←btθt←θt−1−γgtreturn θt\begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \:…

related reading