New Discovery: Retrieval Heads for Long Context

New study by MIT and Peking Univ discovers a new element in transformers: Retrieval Heads!
Retrieval heads impact RAG performance, CoT reasoning and long context length retrieval quality.

These special attention heads are performing the information retrieval of long context, and show exceptional characteristics. Applications are: New dev for better RAG systems, better long context windows retrieval plus improved causal reasoning, better Chain-of-Thought reasoning (CoT), reduce factual hallucination of RAG and LLM by advanced retrieval heads, all induced by these …

applications attention context dev discovery element impact information mit performance quality rag reasoning retrieval show study systems the information transformers