Massive Activations in Large Language Models

arXiv:2402.17762v1 Announce Type: cross
Abstract: We observe an empirical phenomenon in Large Language Models (LLMs) -- very few activations exhibit significantly larger values than others (e.g., 100,000 times larger). We call them massive activations. First, we demonstrate the widespread existence of massive activations across various LLMs and characterize their locations. Second, we find their values largely stay constant regardless of the input, and they function as indispensable bias terms in LLMs. Third, these massive activations lead to the concentration of …

abstract arxiv call cs.cl cs.lg language language models large language large language models llms locations massive observe them type values