Emergent Abilities in Reduced-Scale Generative Language Models

arXiv:2404.02204v1 Announce Type: cross
Abstract: Large language models can solve new tasks without task-specific fine-tuning. This ability, also known as in-context learning (ICL), is considered an emergent ability and is primarily seen in large language models with billions of parameters. This study investigates if such emergent properties are strictly tied to model size or can be demonstrated by smaller models trained on reduced-scale data. To explore this, we simplify pre-training data and pre-train 36 causal language models with parameters varying …

abstract arxiv context cs.cl cs.lg fine-tuning generative in-context learning language language models large language large language models parameters scale solve study tasks type