Initialization Bias of Fourier Neural Operator: Revisiting the Edge of Chaos

arXiv:2310.06379v2 Announce Type: replace
Abstract: This paper investigates the initialization bias of the Fourier neural operator (FNO). A mean-field theory for FNO is established, analyzing the behavior of the random FNO from an \emph{edge of chaos} perspective. We uncover that the forward and backward propagation behaviors exhibit characteristics unique to FNO, induced by mode truncation, while also showcasing similarities to those of densely connected networks. Building upon this observation, we also propose an edge of chaos initialization scheme for FNO …

abstract arxiv behavior bias chaos cs.lg edge fourier mean paper perspective propagation random the edge theory type