Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks. (arXiv:2012.11654v5 [stat.ML] UPDATED)

A recent line of work has analyzed the theoretical properties of deep neural
networks via the Neural Tangent Kernel (NTK). In particular, the smallest
eigenvalue of the NTK has been related to the memorization capacity, the global
convergence of gradient descent algorithms and the generalization of deep nets.
However, existing results either provide bounds in the two-layer setting or
assume that the spectrum of the NTK matrices is bounded away from 0 for
multi-layer networks. In this paper, we provide …

arxiv eigenvalue kernel ml networks relu