On the dynamics of three-layer neural networks: initial condensation

arXiv:2402.15958v1 Announce Type: new
Abstract: Empirical and theoretical works show that the input weights of two-layer neural networks, when initialized with small values, converge towards isolated orientations. This phenomenon, referred to as condensation, indicates that the gradient descent methods tend to spontaneously reduce the complexity of neural networks during the training process. In this work, we elucidate the mechanisms behind the condensation phenomena occurring in the training of three-layer neural networks and distinguish it from the training of two-layer neural …

abstract arxiv complexity converge cs.lg dynamics gradient layer math.ds networks neural networks reduce show small training type values