Vanilla Transformers are Transfer Capability Teachers

arXiv:2403.01994v1 Announce Type: new
Abstract: Recently, Mixture of Experts (MoE) Transformers have garnered increasing attention due to their advantages in model capacity and computational efficiency. However, studies have indicated that MoE Transformers underperform vanilla Transformers in many downstream tasks, significantly diminishing the practical value of MoE models. To explain this issue, we propose that the pre-training performance and transfer capability of a model are joint determinants of its downstream task performance. MoE models, in comparison to vanilla models, have poorer …

abstract advantages arxiv attention capability capacity computational cs.cl efficiency experts issue mixture of experts moe practical studies tasks teachers transfer transformers type value