Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training

arXiv:2405.03133v1 Announce Type: cross
Abstract: Mixture-of-experts (MoE) models facilitate efficient scaling; however, training the router network introduces the challenge of optimizing a non-differentiable, discrete objective. Recently, a fully-differentiable MoE architecture, SMEAR, was proposed (Muqeeth et al., 2023), which softly merges experts in the parameter space; nevertheless, its effectiveness was only demonstrated in downstream fine-tuning on classification tasks. In this paper, we present Lory, the first approach that scales such architectures to autoregressive language model pre-training. Lory introduces two key techniques: …

abstract architecture arxiv autoregressive challenge cs.cl cs.lg differentiable experts however language language model moe network pre-training scaling space training type