Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization. (arXiv:2205.10217v1 [stat.ML])

The Neural Tangent Kernel (NTK) has emerged as a powerful tool to provide
memorization, optimization and generalization guarantees in deep neural
networks. A line of work has studied the NTK spectrum for two-layer and deep
networks with at least a layer with $\Omega(N)$ neurons, $N$ being the number
of training samples. Furthermore, there is increasing evidence suggesting that
deep networks with sub-linear layer widths are powerful memorizers and
optimizers, as long as the number of parameters exceeds the number of …

arxiv ml networks neural networks optimization