Routers in Vision Mixture of Experts: An Empirical Study

arXiv:2401.15969v2 Announce Type: replace-cross
Abstract: Mixture-of-Experts (MoE) models are a promising way to scale up model capacity without significantly increasing computational cost. A key component of MoEs is the router, which decides which subset of parameters (experts) process which feature embeddings (tokens). In this paper, we present a comprehensive study of routers in MoEs for computer vision tasks. We introduce a unified MoE formulation that subsumes different MoEs with two parametric routing tensors. This formulation covers both sparse MoE, which …

abstract arxiv capacity computational cost cs.ai cs.cv cs.lg embeddings experts feature key mixture of experts moe paper parameters process routers scale study tokens type vision