Imposing Gaussian Pre-Activations in a Neural Network. (arXiv:2205.12379v1 [cs.LG])

The goal of the present work is to propose a way to modify both the
initialization distribution of the weights of a neural network and its
activation function, such that all pre-activations are Gaussian. We propose a
family of pairs initialization/activation, where the activation functions span
a continuum from bounded functions (such as Heaviside or tanh) to the identity
function.

This work is motivated by the contradiction between existing works dealing
with Gaussian pre-activations: on one side, the works in …

arxiv network neural network