IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact

arXiv:2403.01241v1 Announce Type: new
Abstract: Large language models (LLMs) excel in natural language processing but demand intensive computation. To mitigate this, various quantization methods have been explored, yet they compromise LLM performance. This paper unveils a previously overlooked type of outlier in LLMs. Such outliers are found to allocate most of the attention scores on initial tokens of input, termed as pivot tokens, which is crucial to the performance of quantized LLMs. Given that, we propose IntactKV to generate the …

abstract arxiv computation cs.ai cs.cl demand excel found language language model language models language processing large language large language model large language models llm llm performance llms natural natural language natural language processing outlier outliers paper performance pivot processing quantization tokens type