Long Context Alignment with Short Instructions and Synthesized Positions

arXiv:2405.03939v1 Announce Type: new
Abstract: Effectively handling instructions with extremely long context remains a challenge for Large Language Models (LLMs), typically necessitating high-quality long data and substantial computational resources. This paper introduces Step-Skipping Alignment (SkipAlign), a new technique designed to enhance the long-context capabilities of LLMs in the phase of alignment without the need for additional efforts beyond training with original data length. SkipAlign is developed on the premise that long-range dependencies are fundamental to enhancing an LLM's capacity of …

abstract alignment arxiv capabilities challenge computational context cs.cl data language language models large language large language models llms paper quality resources synthesized type