CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models

arXiv:2404.08763v1 Announce Type: new
Abstract: Large Language Models (LLMs) have dramatically advanced AI applications, yet their deployment remains challenging due to their immense inference costs. Recent studies ameliorate the computational costs of LLMs by increasing their activation sparsity but suffer from significant performance degradation on downstream tasks. In this work, we introduce a new framework for sparsifying the activations of base LLMs and reducing inference costs, dubbed Contextually Aware Thresholding for Sparsity (CATS). CATS is relatively simple, easy to implement, …

abstract advanced advanced ai ai applications applications arxiv cats computational costs cs.cl cs.lg deployment inference inference costs language language models large language large language models llms performance sparsity studies tasks thresholding type work