Feature Learning and Generalization in Deep Networks with Orthogonal Weights

arXiv:2310.07765v2 Announce Type: replace-cross
Abstract: Fully-connected deep neural networks with weights initialized from independent Gaussian distributions can be tuned to criticality, which prevents the exponential growth or decay of signals propagating through the network. However, such networks still exhibit fluctuations that grow linearly with the depth of the network, which may impair the training of networks with width comparable to depth. We show analytically that rectangular networks with tanh activations and weights initialized from the ensemble of orthogonal matrices have …

abstract arxiv cs.lg feature growth hep-ph hep-th however independent network networks neural networks replace stat.ml through type