In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness

arXiv:2402.11639v1 Announce Type: new
Abstract: A striking property of transformers is their ability to perform in-context learning (ICL), a machine learning framework in which the learner is presented with a novel context during inference implicitly through some data, and tasked with making a prediction in that context. As such that learner must adapt to the context without additional training. We explore the role of softmax attention in an ICL setting where each context encodes a regression task. We show that …

abstract arxiv attention context cs.lg data framework function in-context learning inference machine machine learning making novel prediction property softmax through transformers type