Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability

arXiv:2310.08049v3 Announce Type: replace
Abstract: What is the relationship between model architecture and the ability to perform in-context learning? In this empirical study, we take the first steps toward answering this question. We evaluate thirteen model architectures capable of causal language modeling across a suite of synthetic in-context learning tasks. These selected architectures represent a broad range of paradigms, including recurrent and convolution-based neural networks, transformers, state space model inspired, and other emerging attention alternatives. We discover that all the …

abstract architecture architectures arxiv attention causal context cs.lg in-context learning language modeling question relationship study type