Many-Shot In-Context Learning

arXiv:2404.11018v1 Announce Type: cross
Abstract: Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, without any weight updates. Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples -- the many-shot regime. Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks. While promising, many-shot ICL can be bottlenecked by the available amount of …

abstract arxiv context context windows cs.ai cs.cl cs.lg examples excel few-shot in-context learning inference language language models large language large language models llms observe type updates windows