StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses

arXiv:2403.08312v1 Announce Type: new
Abstract: Standard Large Language Models (LLMs) struggle with handling dialogues with long contexts due to efficiency and consistency issues. According to our observation, dialogue contexts are highly structured, and the special token of \textit{End-of-Utterance} (EoU) in dialogues has the potential to aggregate information. We refer to the EoU tokens as ``conversational attention sinks'' (conv-attn sinks). Accordingly, we introduce StreamingDialogue, which compresses long dialogue history into conv-attn sinks with minimal losses, and thus reduces computational complexity quadratically …

abstract arxiv compression context cs.ai cs.cl dialogue efficiency information language language models large language large language models llms losses observation standard struggle token type via