Adam — PyTorch 2.4 documentation

lr (float, Tensor, optional) – learning rate (default: 1e-3). A tensor LR is not yet supported for all our implementations. Please use a float LR if you are not also specifying fused=True or capturable=True. betas (Tuple[float, float], optional) – coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional) – weight decay (L2 penalty) (default: 0) amsgrad (bool, optional) – whether to use the AMSGrad variant of this algorithm from the paper On the Convergence of Adam and Beyond (default: False) foreach (bool, optional) – whether foreach implementation of optimizer is used. If unspecified by the user (so foreach is None), we will try to use foreach over the for-loop implementation on CUDA, since it is usually significantly more performant. Note that the foreach implementation uses ~ sizeof(params) more peak memory than

Redirecting… Continue to ../../2.13/generated/torch.optim.Adam.html

related reading

• Redirecting…pytorch.org
• Redirecting…pytorch.org
• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• Optimization - PyTorch Lightning 2.6.1 documentationlightning.ai
• the bug that taught me more about PyTorch than years of using it | Elana Simonelanapearl.github.io
• Learning PyTorch with Examples — PyTorch Tutorials 2.12.0+cu130 documentationpytorch.org
• Redirecting…pytorch.org
• AdaGrad - Cornell University Computational Optimization Open Textbook - Optimization Wikioptimization.cbe.cornell.edu
• The Annotated Transformernlp.seas.harvard.edu
• Annotated Research Paper Implementations: Transformers, StyleGAN, Stable Diffusion, DDPM/DDIM, LayerNorm, Nucleus Sampling and morenn.labml.ai
• [2505.21829] In Search of Adam's Secret Saucearxiv.org
• PyTorch internals : ezyang's blogblog.ezyang.com