Optimization — PyTorch Lightning 2.4.0 documentation

lightning.ai · 2,830 words · saved by 1 readers

For the majority of research cases, automatic optimization will do the right thing for you and it is what most users should use. For more advanced use cases like multiple optimizers, esoteric optimization schedules or techniques, use manual optimization. For advanced research topics like reinforcement learning, sparse coding, or GAN research, it may be desirable to manually manage the optimization process, especially when dealing with multiple optimizers at the same time. In this mode, Lightning will handle only accelerator, precision and strategy logic. The users are left with optimizer.zero_grad(), gradient accumulation, optimizer toggling, etc.. To manually optimize, do the following: Set self.automatic_optimization=False in your LightningModule’s __init__. Use the following functions and call them manually: self.optimizers() to access your optimizers (one or multiple) optimizer.zero_grad() to clear the gradients from the previous training step self.manual_backward(loss) instead of

Optimization ¶ Lightning offers two modes for managing the optimization process: Manual Optimization Automatic Optimization For the majority of research cases, automatic optimization will do the right thing for you and it is what most users should use. For more advanced use cases like multiple optimizers, esoteric optimization schedules or techniques, use manual optimization . Manual Optimization ¶ For advanced research topics like reinforcement learning, sparse coding, or GAN research, it may be desirable to manually manage the optimization process, especially when dealing with multiple optim

Optimization — PyTorch Lightning 2.4.0 documentation

related reading