Grokking as the Transition from Lazy to Rich Training Dynamics

arXiv:2310.06110v3 Announce Type: replace-cross
Abstract: We propose that the grokking phenomenon, where the train loss of a neural network decreases much earlier than its test loss, can arise due to a neural network transitioning from lazy training dynamics to a rich, feature learning regime. To illustrate this mechanism, we study the simple setting of vanilla gradient descent on a polynomial regression problem with a two layer neural network which exhibits grokking without regularization in a way that cannot be explained …

abstract arxiv cond-mat.dis-nn cs.lg dynamics feature lazy loss network neural network stat.ml study test train training transition type