In-Context Learning Learns Label Relationships but Is Not Conventional Learning

arXiv:2307.12375v4 Announce Type: replace-cross
Abstract: The predictions of Large Language Models (LLMs) on downstream tasks often improve significantly when including examples of the input--label relationship in the context. However, there is currently no consensus about how this in-context learning (ICL) ability of LLMs works. For example, while Xie et al. (2021) liken ICL to a general-purpose learning algorithm, Min et al. (2022) argue ICL does not even learn label relationships from in-context examples. In this paper, we provide novel insights …

abstract arxiv consensus context cs.ai cs.cl cs.lg example examples however in-context learning language language models large language large language models llms predictions relationship relationships tasks type