Learning in PINNs: Phase transition, total diffusion, and generalization

arXiv:2403.18494v1 Announce Type: new
Abstract: We investigate the learning dynamics of fully-connected neural networks through the lens of gradient signal-to-noise ratio (SNR), examining the behavior of first-order optimizers like Adam in non-convex objectives. By interpreting the drift/diffusion phases in the information bottleneck theory, focusing on gradient homogeneity, we identify a third phase termed ``total diffusion", characterized by equilibrium in the learning rates and homogeneous gradients. This phase is marked by an abrupt SNR increase, uniform residuals across the sample space …

abstract adam arxiv behavior cs.lg diffusion drift dynamics gradient identify information networks neural networks noise signal the information theory through total transition type