[D] On initialization schemes for MLPs: practice and theory

Web: https://www.reddit.com/r/MachineLearning/comments/sdjqab/d_on_initialization_schemes_for_mlps_practice_and/

In this post I am thinking only about MLPs with ReLU activation function.

The default pytorch initialization for linear layers is from a uniform distribution centered at 0 whose limits values depends on the input dimension. Many papers assume initialization from a Gaussian distribution with 0 mean and provide a certain variance.

There is also this work [Pennington (2017)] that proposes orthogonal initialization to achieve what they call dynamical isometry which means that the input-output Jacobian is 1 (or stays …

machinelearning theory