Less is KEN: a Universal and Simple Non-Parametric Pruning Algorithm for Large Language Models

Neural network pruning has become increasingly crucial due to the complexity of neural network models and their widespread use in various fields. Existing pruning algorithms often suffer from limitations such as architecture specificity, excessive complexity and reliance on complex calculations, rendering them impractical for real-world applications. In this paper, we propose KEN: a straightforward, universal and unstructured pruning algorithm based on Kernel Density Estimation (KDE). KEN aims to construct optimized transformer models by selectively preserving the most significant parameters while …

algorithm algorithms applications architecture become complexity cs.lg fields language language models large language large language models limitations network neural network non-parametric parametric pruning reliance rendering simple specificity them world