Dependency-Aware Semi-Structured Sparsity of GLU Variants in Large Language Models

arXiv:2405.01943v1 Announce Type: cross
Abstract: The rapid advancement in Large Language Models (LLMs) has markedly enhanced the capabilities of language understanding and generation. However, the substantial model size poses hardware challenges, affecting both memory size for serving and inference latency for token generation. To address those challenges, we propose Dependency-aware Semi-structured Sparsity (DaSS), a novel method for the recent prevalent SwiGLU-based LLMs pruning. Our approach incorporates structural dependency into the weight magnitude-based unstructured pruning. We introduce an MLP-specific pruning metric …

abstract advancement arxiv capabilities challenges cs.ai cs.cl cs.lg hardware however inference inference latency language language models language understanding large language large language models latency llms memory sparsity token type understanding variants