Parallel Structures in Pre-training Data Yield In-Context Learning

arXiv:2402.12530v1 Announce Type: cross
Abstract: Pre-trained language models (LMs) are capable of in-context learning (ICL): they can adapt to a task with only a few examples given in the prompt without any parameter update. However, it is unclear where this capability comes from as there is a stark distribution shift between pre-training text and ICL prompts. In this work, we study what patterns of the pre-training data contribute to ICL. We find that LMs' ICL ability depends on $\textit{parallel structures}$ …

abstract adapt arxiv capability context cs.ai cs.cl cs.lg data distribution examples in-context learning language language models lms pre-training prompt shift the prompt training training data type update