Nonlinear spiked covariance matrices and signal propagation in deep neural networks

arXiv:2402.10127v1 Announce Type: cross
Abstract: Many recent works have studied the eigenvalue spectrum of the Conjugate Kernel (CK) defined by the nonlinear feature map of a feedforward neural network. However, existing results only establish weak convergence of the empirical eigenvalue distribution, and fall short of providing precise quantitative characterizations of the ''spike'' eigenvalues and eigenvectors that often capture the low-dimensional signal structure of the learning problem. In this work, we characterize these signal eigenvalues and eigenvectors for a nonlinear version …

abstract arxiv convergence covariance cs.lg distribution eigenvalue feature kernel map math.pr math.st network networks neural network neural networks propagation quantitative signal spectrum stat.ml stat.th type