✳flâneur — a map of the web's best reading

Vanishing gradient problem - Wikipedia

en.wikipedia.org · 4,838 words · saved by 1 readers

In machine learning, the vanishing gradient problem is the problem of greatly diverging gradient magnitudes between earlier and later layers encountered when training neural networks with backpropagation. In such methods, neural network weights are updated proportional to their partial derivative of the loss function.[1] As the number of forward propagation steps in a network increases, for instance due to greater network depth, the gradients of earlier weights are calculated with increasingly many multiplications. These multiplications shrink the gradient magnitude. Consequently, the gradients of earlier weights will be exponentially smaller than the gradients of later weights. This difference in gradient magnitude might introduce instability in the training process, slow it, or halt it entirely.[1] For instance, consider the hyperbolic tangent activation function. The gradients of this function are in range [-1,1]. The product of repeated multiplication with such gradients decreases

Vanishing gradient problem - Wikipedia Jump to content From Wikipedia, the free encyclopedia Machine learning model training problem Part of a series on Machine learning and data mining Paradigms Supervised learning Unsupervised learning Semi-supervised learning Self-supervised learning Reinforcement learning Meta-learning Online learning Batch learning Curriculum learning Rule-based learning Neuro-symbolic AI Neuromorphic engineering Quantum machine learning Problems Classification Generative modeling Regression Clustering Dimensionality reduction Density estimation Anomaly detection Data cle

Explore this link on the map →

related reading