Three Mechanisms of Feature Learning in the Exact Solution of a Latent Variable Model

arXiv:2401.07085v2 Announce Type: replace
Abstract: We identify and exactly solve the learning dynamics of a one-hidden-layer linear model at any finite width whose limits exhibit both the kernel phase and the feature learning phase. We analyze the phase diagram of this model in different limits of common hyperparameters including width, layer-wise learning rates, scale of output, and scale of initialization. Our solution identifies three novel prototype mechanisms of feature learning: (1) learning by alignment, (2) learning by disalignment, and (3) …

abstract analyze arxiv cs.ai cs.lg dynamics feature hidden identify kernel latent variable model layer linear linear model solution solve type