Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models

arXiv:2404.02837v1 Announce Type: new
Abstract: This paper reveals the phenomenon of parameter heterogeneity in large language models (LLMs). We find that a small subset of ``cherry'' parameters exhibit a disproportionately large influence on model performance, while the vast majority of parameters have minimal impact. This heterogeneity is found to be prevalent across different model families, scales, and types. Motivated by this observation, we propose CherryQ, a novel quantization method that unifies the optimization of mixed-precision parameters. CherryQ identifies and preserves …

abstract arxiv cherry cs.cl found impact influence language language models large language large language models llms paper parameters performance quantization small type vast