Geometry and Dynamics of LayerNorm

arXiv:2405.04134v1 Announce Type: new
Abstract: A technical note aiming to offer deeper intuition for the LayerNorm function common in deep neural networks. LayerNorm is defined relative to a distinguished 'neural' basis, but it does more than just normalize the corresponding vector elements. Rather, it implements a composition -- of linear projection, nonlinear scaling, and then affine transformation -- on input activation vectors. We develop both a new mathematical expression and geometric intuition, to make the net effect more transparent. We …

abstract arxiv cs.lg dynamics function geometry intuition linear networks neural networks projection scaling technical type vector