Rectifying Demonstration Shortcut in In-Context Learning

arXiv:2403.09488v1 Announce Type: new
Abstract: Large language models (LLMs) are able to solve various tasks with only a few demonstrations utilizing their in-context learning (ICL) abilities. However, LLMs often rely on their pre-trained semantic priors of demonstrations rather than on the input-label relationships to proceed with ICL prediction. In this work, we term this phenomenon as the `Demonstration Shortcut'. While previous works have primarily focused on improving ICL prediction results for predefined tasks, we aim to rectify the Demonstration Shortcut, …

abstract arxiv context cs.ai cs.cl however in-context learning language language models large language large language models llms prediction relationships semantic shortcut solve tasks type work