An exactly solvable model for emergence and scaling laws

arXiv:2404.17563v1 Announce Type: new
Abstract: Deep learning models can exhibit what appears to be a sudden ability to solve a new problem as training time ($T$), training data ($D$), or model size ($N$) increases, a phenomenon known as emergence. In this paper, we present a framework where each new ability (a skill) is represented as a basis function. We solve a simple multi-linear model in this skill-basis, finding analytic expressions for the emergence of new skills, as well as for …

abstract arxiv cond-mat.dis-nn cs.lg data deep learning emergence framework laws paper scaling solve stat.ml training training data type