Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy

arXiv:2310.01334v2 Announce Type: replace
Abstract: Sparsely activated Mixture-of-Experts (SMoE) has shown promise to scale up the learning capacity of neural networks, however, they have issues like (a) High Memory Usage, due to duplication of the network layers into multiple copies as experts; and (b) Redundancy in Experts, as common learning-based routing policies suffer from representational collapse. Therefore, vanilla SMoE models are memory inefficient and non-scalable, especially for resource-constrained downstream scenarios. In this paper, we ask: Can we craft a compact …

abstract arxiv capacity cs.ai cs.cl cs.lg experts however memory merge multiple network networks neural networks policy redundancy routing scale type usage