Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks

arXiv:2305.17212v3 Announce Type: replace
Abstract: This study investigates how weight decay affects the update behavior of individual neurons in deep neural networks through a combination of applied analysis and experimentation. Weight decay can cause the expected magnitude and angular updates of a neuron's weight vector to converge to a steady state we call rotational equilibrium. These states can be highly homogeneous, effectively balancing the average rotation -- a proxy for the effective learning rate -- across different layers and neurons. …

abstract analysis angular arxiv behavior combination converge cs.lg equilibrium experimentation individual neurons networks neural networks neuron neurons study through type update updates vector