How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?

arXiv:2310.08391v2 Announce Type: replace-cross
Abstract: Transformers pretrained on diverse tasks exhibit remarkable in-context learning (ICL) capabilities, enabling them to solve unseen tasks solely based on input contexts without adjusting model parameters. In this paper, we study ICL in one of its simplest setups: pretraining a linearly parameterized single-layer linear attention model for linear regression with a Gaussian prior. We establish a statistical task complexity bound for the attention model pretraining, showing that effective pretraining only requires a small number of …

abstract adjusting arxiv capabilities context cs.lg diverse enabling in-context learning layer linear linear regression paper parameters pretraining regression solve stat.ml study tasks them transformers type