Unifying Low Dimensional Observations in Deep Learning Through the Deep Linear Unconstrained Feature Model

arXiv:2404.06106v1 Announce Type: new
Abstract: Modern deep neural networks have achieved high performance across various tasks. Recently, researchers have noted occurrences of low-dimensional structure in the weights, Hessian's, gradients, and feature vectors of these networks, spanning different datasets and architectures when trained to convergence. In this analysis, we theoretically demonstrate these observations arising, and show how they can be unified within a generalized unconstrained feature model that can be considered analytically. Specifically, we consider a previously described structure called Neural …

abstract architectures arxiv convergence cs.lg datasets deep learning feature linear low modern networks neural networks performance researchers tasks through type vectors