ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models

arXiv:2402.13516v1 Announce Type: new
Abstract: Activation sparsity refers to the existence of considerable weakly-contributed elements among activation outputs. As a prevalent property of the models using the ReLU activation function, it has been proven a promising paradigm to boost model inference efficiency. Nevertheless, most large language models (LLMs) adopt activation functions without intrinsic activation sparsity (e.g., GELU and Swish). Some recent efforts have explored introducing ReLU or its variants as the substitutive activation function to help LLMs achieve activation sparsity …

abstract arxiv boost contributed cs.ai cs.cl cs.lg efficiency function inference intrinsic language language models large language large language models llms paradigm property relu sparsity type