flâneur

Grokking (machine learning) - Wikipedia

en.wikipedia.org · 1,180 words · saved by 1 readers

In machine learning, grokking, or delayed generalization, is a transition to generalization that occurs many training iterations after the interpolation threshold, after many iterations of seemingly little progress, as opposed to the usual process where generalization occurs slowly and progressively once the interpolation threshold has been reached.[2][3][4] Grokking was introduced in January 2022 by OpenAI researchers investigating how neural network perform calculations. It is derived from the word grok coined by Robert Heinlein in his novel Stranger in a Strange Land.[1] Grokking can be understood as a phase transition during the training process.[5] In particular, recent work has shown that grokking may be due to a complexity phase transition in the model during training.[6] While grokking has been thought of as largely a phenomenon of relatively shallow models, grokking has been observed in deep neural networks and non-neural models and is the subject of active research.[7][8][9][

Grokking (machine learning) - Wikipedia Jump to content From Wikipedia, the free encyclopedia Phase transition in machine learning Not to be confused with Grok (chatbot) . The blue loss curves represent early memorization of the training set ( overfitting ), and the red curves show late generalization, with the learning of a modular addition algorithm that works with unseen inputs. [ 1 ] In machine learning , grokking , or delayed generalization , is a phenomenon observed in some settings where a model abruptly transitions from overfitting (performing well only on training data ) to generalizi

related reading