Fourier Circuits in Neural Networks: Unlocking the Potential of Large Language Models in Mathematical Reasoning and Modular Arithmetic

arXiv:2402.09469v1 Announce Type: new
Abstract: In the evolving landscape of machine learning, a pivotal challenge lies in deciphering the internal representations harnessed by neural networks and Transformers. Building on recent progress toward comprehending how networks execute distinct target functions, our study embarks on an exploration of the underlying reasons behind networks adopting specific computational strategies. We direct our focus to the complex algebraic learning task of modular addition involving $k$ inputs. Our research presents a thorough analytical characterization of the …

abstract arxiv building challenge cs.lg fourier functions landscape language language models large language large language models lies machine machine learning mathematical reasoning modular networks neural networks pivotal progress reasoning stat.ml study transformers type