Memory capacity of two layer neural networks with smooth activations

arXiv:2308.02001v3 Announce Type: replace
Abstract: Determining the memory capacity of two layer neural networks with $m$ hidden neurons and input dimension $d$ (i.e., $md+2m$ total trainable parameters), which refers to the largest size of general data the network can memorize, is a fundamental machine learning question. For activations that are real analytic at a point and, if restricting to a polynomial there, have sufficiently high degree, we establish a lower bound of $\lfloor md/2\rfloor$ and optimality up to a factor …

abstract arxiv capacity cs.lg data fundamental general hidden layer machine machine learning math.oc memory network networks neural networks neurons parameters question total type