Infinite Context Windows? LLMs for Streaming Applications with Attention Sinks

In recent years, natural language processing has been revolutionized by the advent of large language models (LLMs). Massive neural networks like GPT-3, PaLM, and BlenderBot have demonstrated remarkable proficiency at various language tasks like conversational AI, summarization, and question-answering. However, a major impediment restricts their practical deployment in real-world streaming applications.

LLMs are pre-trained on texts of finite lengths, usually a few thousand tokens. As a result, their performance deteriorates rapidly when presented with sequence lengths exceeding their training corpus. …

ai applications attention blenderbot context context windows conversational conversational ai deployment discuss gpt gpt-3 language language models language processing large language large language models llms major massive natural natural language natural language processing networks neural networks palm practical processing programming streaming summarization tasks windows world