SADMoE: Exploiting Activation Sparsity with Dynamic-k Gating

arXiv:2310.04361v2 Announce Type: replace
Abstract: Transformer models, despite their impressive performance, often face practical limitations due to their high computational requirements. At the same time, such models exhibit significant activation sparsity, which can be leveraged to reduce the inference cost by transforming parts of the network into Mixture-of-Experts (MoE) layers. However, despite the crucial role of activation sparsity, its impact on this process remains unexplored. In this paper, we enhance the efficiency of MoE conversion through activation sparsity enforcement. Moreover, …

abstract arxiv computational cost cs.lg dynamic experts face inference limitations moe network performance practical reduce requirements sparsity transformer transformer models type