Linearly Constrained Weights: Reducing Activation Shift for Faster Training of Neural Networks

arXiv:2403.13833v1 Announce Type: cross
Abstract: In this paper, we first identify activation shift, a simple but remarkable phenomenon in a neural network in which the preactivation value of a neuron has non-zero mean that depends on the angle between the weight vector of the neuron and the mean of the activation vector in the previous layer. We then propose linearly constrained weights (LCW) to reduce the activation shift in both fully connected and convolutional layers. The impact of reducing the …

abstract arxiv cs.lg cs.ne faster identify mean network networks neural network neural networks neuron paper shift simple stat.ml training type value vector