Training Nonlinear Transformers for Efficient In-Context Learning: A Theoretical Learning and Generalization Analysis

arXiv:2402.15607v1 Announce Type: new
Abstract: Transformer-based large language models have displayed impressive in-context learning capabilities, where a pre-trained model can handle new tasks without fine-tuning by simply augmenting the query with some input-output examples from that task. Despite the empirical success, the mechanics of how to train a Transformer to achieve ICL and the corresponding ICL capacity is mostly elusive due to the technical challenges of analyzing the nonconvex training problems resulting from the nonlinear self-attention and nonlinear activation in …

abstract analysis arxiv capabilities context cs.lg examples fine-tuning in-context learning input-output language language models large language large language models query success tasks train training transformer transformers type