Progressive Feedforward Collapse of ResNet Training

arXiv:2405.00985v1 Announce Type: new
Abstract: Neural collapse (NC) is a simple and symmetric phenomenon for deep neural networks (DNNs) at the terminal phase of training, where the last-layer features collapse to their class means and form a simplex equiangular tight frame aligning with the classifier vectors. However, the relationship of the last-layer features to the data and intermediate layers during training remains unexplored. To this end, we characterize the geometry of intermediate layers of ResNet and propose a novel conjecture, …

abstract arxiv class classifier cs.ai cs.lg features form however layer math.oc math.st networks neural collapse neural networks relationship resnet simple stat.th terminal training type vectors