Compute Trends Across Three Eras of Machine Learning. (arXiv:2202.05924v2 [cs.LG] UPDATED)

Compute, data, and algorithmic advances are the three fundamental factors
that guide the progress of modern Machine Learning (ML). In this paper we study
trends in the most readily quantified factor - compute. We show that before
2010 training compute grew in line with Moore's law, doubling roughly every 20
months. Since the advent of Deep Learning in the early 2010s, the scaling of
training compute has accelerated, doubling approximately every 6 months. In
late 2015, a new trend emerged …

arxiv compute learning machine machine learning trends