DEEP-ICL: Definition-Enriched Experts for Language Model In-Context Learning

arXiv:2403.04233v1 Announce Type: new
Abstract: It has long been assumed that the sheer number of parameters in large language models (LLMs) drives in-context learning (ICL) capabilities, enabling remarkable performance improvements by leveraging task-specific demonstrations. Challenging this hypothesis, we introduce DEEP-ICL, a novel task Definition Enriched ExPert Ensembling methodology for ICL. DEEP-ICL explicitly extracts task definitions from given demonstrations and generates responses through learning task-specific examples. We argue that improvement from ICL does not directly rely on model size, but essentially …

abstract arxiv capabilities context cs.ai cs.cl definition enabling expert experts hypothesis improvements in-context learning language language model language models large language large language models llms methodology novel parameters performance type