✳flâneur — a map of the web's best reading

Choosing Gradient Norm Clip Value? [D] : r/MachineLearning

reddit.com · 6 words · saved by 1 readers

How do I choose the max value to use for global gradient norm clipping? The value must somehow depend on the number of parameters because more parameters means the parameter gradient vector has more numbers in it and higher dimensional vectors have bigger norms than lower dimensional ones. I can not find anything online giving advice on choosing the max norm value. Can anyone give me advice that improves over "just try loads of values". (Am training transformers) [sidequest] Legends are told about weird results when adam is combined with gradient clipping. We don't have compute and I have cannot leave my sick brother to go on an adventure. If only there was someone who could look into that old myth ... 😂 😂 😂 I wrote a paper a few months back about adapting the clipping value based on the history of gradient norms using percentiles. I use it all the time now for every model I train (with p=10): https://github.com/pseeth/autoclip. Give it a shot! For my use cases (training audio netwo

Reddit - Please wait for verification

Explore this link on the map →

related reading