Revisiting the Hypothesis: Do pretrained Transformers Learn In-Context by Gradient Descent?

arXiv:2310.08540v4 Announce Type: replace-cross
Abstract: The emergence of In-Context Learning (ICL) in LLMs remains a significant phenomenon with little understanding. To explain ICL, recent studies try to theoretically connect it to Gradient Descent (GD). We ask, does this connection hold up in actual pre-trained models?
We highlight the limiting assumptions in prior works that make their context considerably different from the practical context in which language models are trained. For example, the theoretical hand-constructed weights used in these studies have …

abstract arxiv context cs.ai cs.cl cs.lg emergence gradient highlight hypothesis in-context learning learn llms pre-trained models studies transformers type understanding