Simplicity Bias of Transformers to Learn Low Sensitivity Functions

arXiv:2403.06925v1 Announce Type: new
Abstract: Transformers achieve state-of-the-art accuracy and robustness across many tasks, but an understanding of the inductive biases that they have and how those biases are different from other neural network architectures remains elusive. Various neural network architectures such as fully connected networks have been found to have a simplicity bias towards simple functions of the data; one version of this simplicity bias is a spectral bias to learn simple functions in the Fourier space. In this …

abstract accuracy architectures art arxiv bias biases cs.ai cs.cl cs.lg found functions inductive learn low network networks neural network robustness sensitivity simplicity state stat.ml tasks transformers type understanding