Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs

arXiv:2404.10308v1 Announce Type: new
Abstract: Large language models (LLMs) have shown remarkable performance in various natural language processing tasks. However, a primary constraint they face is the context limit, i.e., the maximum number of tokens they can process. Previous works have explored architectural changes and modifications in positional encoding to relax the constraint, but they often require expensive training or do not address the computational demands of self-attention. In this paper, we present Hierarchical cOntext MERging (HOMER), a new training-free …

arxiv context cs.ai cs.lg hierarchical llms merging type understanding