flâneur

Adam — PyTorch 2.4 documentation

pytorch.org · 2,048 words · saved by 1 readers

lr (float, Tensor, optional) – learning rate (default: 1e-3). A tensor LR is not yet supported for all our implementations. Please use a float LR if you are not also specifying fused=True or capturable=True. betas (Tuple[float, float], optional) – coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional) – weight decay (L2 penalty) (default: 0) amsgrad (bool, optional) – whether to use the AMSGrad variant of this algorithm from the paper On the Convergence of Adam and Beyond (default: False) foreach (bool, optional) – whether foreach implementation of optimizer is used. If unspecified by the user (so foreach is None), we will try to use foreach over the for-loop implementation on CUDA, since it is usually significantly more performant. Note that the foreach implementation uses ~ sizeof(params) more peak memory than

class torch.optim.Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False, *, foreach=None, maximize=False, capturable=False, differentiable=False, fused=None, decoupled_weight_decay=False)[source]# Implements Adam algorithm. input:γ (lr),β1,β2 (betas),θ0 (params),f(θ) (objective)λ (weight decay), amsgrad, maximize, ϵ (epsilon)initialize:m0←0 ( first moment),v0←0 (second moment), v0max←0for t=1 to … doif maximize:gt←−∇θft(θt−1)elsegt←∇θft(θt−1)if λ≠0gt←gt+λθt−1mt←β1mt−1+(1−β1)gtvt←β2vt−1+(1−β2)gt2mt^←mt/(1−β1t)if…

related reading