Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment

arXiv:2405.03594v1 Announce Type: new
Abstract: Large language models (LLMs) have revolutionized Natural Language Processing (NLP), but their size creates computational bottlenecks. We introduce a novel approach to create accurate, sparse foundational versions of performant LLMs that achieve full accuracy recovery for fine-tuning tasks at up to 70% sparsity. We achieve this for the LLaMA-2 7B model by combining the SparseGPT one-shot pruning method and sparse pretraining of those models on a subset of the SlimPajama dataset mixed with a Python …

abstract accuracy arxiv bottlenecks computational create cs.ai cs.cl deployment enabling fine-tuning foundational language language models language processing large language large language models llama llama models llms natural natural language natural language processing nlp novel pretraining processing recovery sparsity tasks type versions