Meta AI LM-Infinite - Massive LLM improvement!

Paper: https://huggingface.co/papers/2308.16137

--- Overview:
This paper identifies and addresses a key limitation of large language models (LLMs) - the inability to generalize to sequence lengths longer than their training corpus. Even models using relative position encodings struggle to generate coherent text beyond contexts seen during training. The authors diagnose three contributing factors through empirical analysis, and propose a simple and efficient solution called LM-Infinite that enables on-the-fly length generalization without retraining. When tested on models like LLaMA and GPT-J, LM-Infinite …

analysis authors beyond language language models large language large language models llms overview paper simple solution text through training