LongHeads: Multi-Head Attention is Secretly a Long Context Processor

arXiv:2402.10685v1 Announce Type: new
Abstract: Large language models (LLMs) have achieved impressive performance in numerous domains but often struggle to process lengthy inputs effectively and efficiently due to limited length generalization and attention's quadratic computational demands. Many sought to mitigate this by restricting the attention window within the pre-trained length. However, these methods introduce new issues such as ignoring the middle context and requiring additional training. To address these problems, we propose LongHeads, a training-free framework that enhances LLM's long …

abstract arxiv attention computational context cs.ai cs.cl domains head inputs language language models large language large language models llms multi-head multi-head attention performance process processor struggle type