Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks

arXiv:2406.02550v1 Announce Type: new
Abstract: Large language models can solve tasks that were not present in the training set. This capability is believed to be due to in-context learning and skill composition. In this work, we study the emergence of in-context learning and skill composition in a collection of modular arithmetic tasks. Specifically, we consider a finite collection of linear modular functions $z = a \, x + b \, y \;\mathrm{mod}\; p$ labeled by the vector $(a, b) \in …

abstract arxiv capability cond-mat.dis-nn context context learning cs.lg emergence grok hep-th in-context learning language language models large language large language models modular set skill solve stat.ml study tasks training type work