neural nets

N/A

96 Jonathan Richard Shewchuk 17 Vanishing Gradients; ReLUs; Output Units and Losses; Neurobiology THE VANISHING GRADIENT PROBLEM; ReLUs [Last lecture, we put a logistic function at the output of every unit except the input units. These units are called sigmoid units. But in practice, sigmoid units are usually a poor choice for hidden layers.] Problem: When unit output s is close to 0 or 1 for most training points, s0 = s(1 s) ⇡ 0, so gradient descent changes s very slowly. Unit is “stuck.” Slow training.…

related reading

• Neural networks and deep learningneuralnetworksanddeeplearning.com
• Yes you should understand backprop | by Andrej Karpathy | Mediumkarpathy.medium.com
• Vanishing gradient problem - Wikipediaen.wikipedia.org
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• The Decade of Deep Learning | Leo Gaobmk.sh
• CS231n Deep Learning for Computer Visioncs231n.github.io
• A Recipe for Training Neural Networkskarpathy.github.io
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• The Little Book of Deep Learningfleuret.org
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• ReLU Activation Function in Deep Learning - GeeksforGeeksgeeksforgeeks.org
• Neural networks and deep learningneuralnetworksanddeeplearning.com