Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context

arXiv:2312.06528v4 Announce Type: replace
Abstract: Many neural network architectures are known to be Turing Complete, and can thus, in principle implement arbitrary algorithms. However, Transformers are unique in that they can implement gradient-based learning algorithms under simple parameter configurations. This paper provides theoretical and empirical evidence that (non-linear) Transformers naturally learn to implement gradient descent in function space, which in turn enable them to learn non-linear functions in context. Our results apply to a broad class of combinations of non-linear …

abstract algorithms architectures arxiv context cs.lg evidence functional functions gradient learn linear network neural network non-linear paper simple transformers turing type