Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models

arXiv:2404.05567v1 Announce Type: new
Abstract: Mixture-of-Experts (MoE) language models can reduce computational costs by 2-4$\times$ compared to dense models without sacrificing performance, making them more efficient in computation-bounded scenarios. However, MoE models generally require 2-4$\times$ times more parameters to achieve comparable performance to a dense model, which incurs larger GPU memory requirements and makes MoE models less efficient in I/O-bounded scenarios like autoregressive generation. In this work, we propose a hybrid dense training and sparse inference framework for MoE models …

abstract arxiv computation computational costs cs.ai cs.cl cs.lg experts gpu however inference language language models making moe parameters performance reduce them training type