Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization

arXiv:2402.12550v1 Announce Type: cross
Abstract: The Mixture of Experts (MoE) paradigm provides a powerful way to decompose inscrutable dense layers into smaller, modular computations often more amenable to human interpretation, debugging, and editability. A major problem however lies in the computational cost of scaling the number of experts to achieve sufficiently fine-grained specialization. In this paper, we propose the Multilinear Mixutre of Experts (MMoE) layer to address this, focusing on vision models. MMoE layers perform an implicit computation on prohibitively …

arxiv cs.cv cs.lg expert experts factorization mixture of experts scalable through type