In-Context Learning State Vector with Inner and Momentum Optimization

arXiv:2404.11225v1 Announce Type: new
Abstract: Large Language Models (LLMs) have exhibited an impressive ability to perform In-Context Learning (ICL) from only a few examples. Recent works have indicated that the functions learned by ICL can be represented through compressed vectors derived from the transformer. However, the working mechanisms and optimization of these vectors are yet to be thoroughly explored. In this paper, we address this gap by presenting a comprehensive analysis of these compressed vectors, drawing parallels to the parameters …

arxiv context cs.ai cs.cl in-context learning optimization state type vector