Linear Transformers are Versatile In-Context Learners

arXiv:2402.14180v1 Announce Type: new
Abstract: Recent research has demonstrated that transformers, particularly linear attention models, implicitly execute gradient-descent-like algorithms on data provided in-context during their forward inference step. However, their capability in handling more complex problems remains unexplored. In this paper, we prove that any linear transformer maintains an implicit linear model and can be interpreted as performing a variant of preconditioned gradient descent. We also investigate the use of linear transformers in a challenging scenario where the training data …

abstract algorithms arxiv attention capability context cs.lg data gradient gradient-descent inference linear linear model paper prove research transformer transformers type