A Simple and Effective Pruning Approach for Large Language Models

arXiv:2306.11695v3 Announce Type: replace-cross
Abstract: As their size increases, Large Languages Models (LLMs) are natural candidates for network pruning methods: approaches that drop a subset of network weights while striving to preserve performance. Existing methods, however, require either retraining, which is rarely affordable for billion-scale LLMs, or solving a weight reconstruction problem reliant on second-order information, which may also be computationally expensive. In this paper, we introduce a novel, straightforward yet effective pruning method, termed Wanda (Pruning by Weights and …

arxiv cs.ai cs.cl cs.lg language language models large language large language models pruning simple type