The lazy (NTK) and rich ($\mu$P) regimes: a gentle tutorial

arXiv:2404.19719v1 Announce Type: new
Abstract: A central theme of the modern machine learning paradigm is that larger neural networks achieve better performance on a variety of metrics. Theoretical analyses of these overparameterized models have recently centered around studying very wide neural networks. In this tutorial, we provide a nonrigorous but illustrative derivation of the following fact: in order to train wide networks effectively, there is only one degree of freedom in choosing hyperparameters such as the learning rate and the …

abstract arxiv cs.lg lazy machine machine learning metrics modern networks neural networks paradigm performance stat.ml studying tutorial type