Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

arXiv:2311.04902v2 Announce Type: replace-cross
Abstract: Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, removing some model weights without hurting performance. Prior approaches such as magnitude pruning, SparseGPT, and Wanda, either concentrated solely on weights or integrated weights with activations for sparsity. However, they overlooked the informative gradients derived from pretrained LLMs. In this paper, we present a novel sparsity-centric pruning method for pretrained LLMs, termed Gradient-based Language Model Pruner (GBLM-Pruner). GBLM-Pruner leverages the first-order …

arxiv beyond cs.ai cs.cl cs.lg decisions language language models large language large language models pruning type