neural nets

N/A

96 Jonathan Richard Shewchuk 17 Vanishing Gradients; ReLUs; Output Units and Losses; Neurobiology THE VANISHING GRADIENT PROBLEM; ReLUs [Last lecture, we put a logistic function at the output of every unit except the input units. These units are called sigmoid units. But in practice, sigmoid units are usually a poor choice for hidden layers.] Problem: When unit output s is close to 0 or 1 for most training points, s0 = s(1 s) ⇡ 0, so gradient descent changes s very slowly. Unit is “stuck.” Slow training.…

related reading

• Neural networks and deep learningneuralnetworksanddeeplearning.com
• neurobio and vanishing gradientspeople.eecs.berkeley.edu
• A Recipe for Training Neural Networkskarpathy.github.io
• Yes you should understand backprop | by Andrej Karpathy | Mediumkarpathy.medium.com
• nn-notes.pdfboris-hanin.github.io
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• Vanishing gradient problem - Wikipediaen.wikipedia.org
• Neural Networks, Manifolds, and Topology -- colah's blogcolah.github.io
• kernelspeople.eecs.berkeley.edu
• A Recipe for Training Neural Networkskarpathy.github.io
• Neural networks and deep learningneuralnetworksanddeeplearning.com