Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference

arXiv:2308.12066v3 Announce Type: replace
Abstract: Large language models (LLMs) based on transformers have made significant strides in recent years, the success of which is driven by scaling up their model size. Despite their high algorithmic performance, the computational and memory requirements of LLMs present unprecedented challenges. To tackle the high compute requirements of LLMs, the Mixture-of-Experts (MoE) architecture was introduced which is able to scale its model size without proportionally scaling up its computational requirements. Unfortunately, MoE's high memory demands …

abstract algorithm arxiv challenges computational cs.ai cs.ar cs.lg design expert inference language language models large language large language models llms memory moe performance requirements scalable scaling scaling up success transformers type