Average gradient outer product as a mechanism for deep neural collapse

arXiv:2402.13728v1 Announce Type: new
Abstract: Deep Neural Collapse (DNC) refers to the surprisingly rigid structure of the data representations in the final layers of Deep Neural Networks (DNNs). Though the phenomenon has been measured in a wide variety of settings, its emergence is only partially understood. In this work, we provide substantial evidence that DNC formation occurs primarily through deep feature learning with the average gradient outer product (AGOP). This takes a step further compared to efforts that explain neural …

abstract arxiv cs.lg data emergence gradient networks neural collapse neural networks product stat.ml type work