Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

arXiv:2403.04797v1 Announce Type: cross
Abstract: This paper aims to overcome the "lost-in-the-middle" challenge of large language models (LLMs). While recent advancements have successfully enabled LLMs to perform stable language modeling with up to 4 million tokens, the persistent difficulty faced by most LLMs in identifying relevant information situated in the middle of the context has not been adequately tackled. To address this problem, this paper introduces Multi-scale Positional Encoding (Ms-PoE) which is a simple yet effective plug-and-play approach to enhance …

arxiv cs.cl cs.lg encoding found language language models positional encoding type via