Web: http://arxiv.org/abs/2110.14163

June 23, 2022, 1:11 a.m. | Rubing Yang, Jialin Mao, Pratik Chaudhari

cs.LG updates on arXiv.org arxiv.org

We show that the input correlation matrix of typical classification datasets
has an eigenspectrum where, after a sharp initial drop, a large number of small
eigenvalues are distributed uniformly over an exponentially large range. This
structure is mirrored in a network trained on this data: we show that the
Hessian and the Fisher Information Matrix (FIM) have eigenvalues that are
spread uniformly over exponentially large ranges. We call such eigenspectra
"sloppy" because sets of weights corresponding to small eigenvalues can …

arxiv capacity data deep deep learning learning lg

More from arxiv.org / cs.LG updates on arXiv.org

Machine Learning Researcher - Saalfeld Lab

@ Howard Hughes Medical Institute - Chevy Chase, MD | Ashburn, Virginia

Project Director, Machine Learning in US Health

@ ideas42.org | Remote, US

Data Science Intern

@ NannyML | Remote

Machine Learning Engineer NLP/Speech

@ Play.ht | Remote

Research Scientist, 3D Reconstruction

@ Yembo | Remote, US

Clinical Assistant or Associate Professor of Management Science and Systems

@ University at Buffalo | Buffalo, NY