Gradient descent induces alignment between weights and the empirical NTK for deep non-linear networks

Understanding the mechanisms through which neural networks extract statistics from input-label pairs is one of the most important unsolved problems in supervised learning. Prior works have identified that the gram matrices of the weights in trained neural networks of general architectures are proportional to the average gradient outer product of the model, in a statement known as the Neural Feature Ansatz (NFA). However, the reason these quantities become correlated during training is poorly understood. In this work, we explain the …

alignment architectures cs.ai cs.lg extract general gradient linear networks neural networks non-linear prior statistics stat.ml supervised learning through understanding unsolved