torch.Tensor.backward — PyTorch 2.4 documentation

The graph is differentiated using the chain rule. If the tensor is non-scalar (i.e. its data has more than one element) and requires gradient, the function additionally requires specifying a gradient. It should be a tensor of matching type and shape, that represents the gradient of the differentiated function w.r.t. self. This function accumulates gradients in the leaves - you might need to zero .grad attributes or set them to None before calling it. See Default gradient layouts for details on the memory layout of accumulated gradients. NOTE If you run any forward ops, create gradient, and/or call backward in a user-specified CUDA stream context, see Stream semantics of backward passes. NOTE When inputs are provided and a given input is not a leaf, the current implementation will call its grad_fn (though it is not strictly needed to get this gradients). It is an implementation detail on which the user should not rely. See https://github.com/pytorch/pytorch/pull/60521#issuecomment-86706

Redirecting… Continue to ../../2.13/generated/torch.Tensor.backward.html

related reading

• PyTorch internals : ezyang's blogblog.ezyang.com
• Yes you should understand backprop | by Andrej Karpathy | Mediumkarpathy.medium.com
• Differentiable Programming from Scratchthenumb.at
• Calculus on Computational Graphs: Backpropagation -- colah's blogcolah.github.io
• Tiny TPUtinytpu.com
• Learning PyTorch with Examples — PyTorch Tutorials 2.12.0+cu130 documentationpytorch.org
• the bug that taught me more about PyTorch than years of using it | Elana Simonelanapearl.github.io
• torch.Tensor — PyTorch 2.12 documentationdocs.pytorch.org
• 2404.17625arxiv.org
• CS231n Deep Learning for Computer Visioncs231n.github.io
• Reverse-mode automatic differentiation from scratch, in Python | sidsitesidsite.com
• microgptkarpathy.github.io