Explaining Neural Scaling Laws

arXiv:2102.06701v2 Announce Type: replace
Abstract: The population loss of trained deep neural networks often follows precise power-law scaling relations with either the size of the training dataset or the number of parameters in the network. We propose a theory that explains the origins of and connects these scaling laws. We identify variance-limited and resolution-limited scaling behavior for both dataset and model size, for a total of four scaling regimes. The variance-limited scaling follows simply from the existence of a well-behaved …

abstract arxiv behavior cond-mat.dis-nn cs.lg dataset identify law laws loss network networks neural networks parameters population power power-law relations resolution scaling stat.ml theory training type variance