Concept-aware Data Construction Improves In-context Learning of Language Models

arXiv:2403.09703v1 Announce Type: new
Abstract: Many recent language models (LMs) are capable of in-context learning (ICL), manifested in the LMs' ability to perform a new task solely from natural-language instruction. Previous work curating in-context learners assumes that ICL emerges from a vast over-parametrization or the scale of multi-task training. However, recent theoretical work attributes the ICL ability to concept-dependent training data and creates functional in-context learners even in small-scale, synthetic settings.
In this work, we practically explore this newly identified …

abstract arxiv concept construction context cs.ai cs.cl data however in-context learning language language models lms natural scale training type vast work