In-context Learning and Gradient Descent Revisited

arXiv:2311.07772v4 Announce Type: replace-cross
Abstract: In-context learning (ICL) has shown impressive results in few-shot learning tasks, yet its underlying mechanism is still not fully understood. A recent line of work suggests that ICL performs gradient descent (GD)-based optimization implicitly. While appealing, much of the research focuses on simplified settings, where the parameters of a shallow model are optimized. In this work, we revisit evidence for ICL-GD correspondence on realistic NLP tasks and models. We find gaps in evaluation, both in …

abstract arxiv context cs.cl cs.lg few-shot few-shot learning gradient in-context learning line optimization parameters research results simplified tasks type work