GradMax: Growing Neural Networks using Gradient Information. (arXiv:2201.05125v1 [cs.LG])

The architecture and the parameters of neural networks are often optimized
independently, which requires costly retraining of the parameters whenever the
architecture is modified. In this work we instead focus on growing the
architecture without requiring costly retraining. We present a method that adds
new neurons during training without impacting what is already learned, while
improving the training dynamics. We achieve the latter by maximizing the
gradients of the new weights and find the optimal initialization efficiently by
means of …

arxiv gradient information networks neural networks