Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study

arXiv:2403.17404v1 Announce Type: new
Abstract: Mixture-of-Experts (MoE) represents an ensemble methodology that amalgamates predictions from several specialized sub-models (referred to as experts). This fusion is accomplished through a router mechanism, dynamically assigning weights to each expert's contribution based on the input data. Conventional MoE mechanisms select all available experts, incurring substantial computational costs. In contrast, Sparse Mixture-of-Experts (Sparse MoE) selectively engages only a limited number, or even just one expert, significantly reducing computation overhead while empirically preserving, and sometimes even …

abstract analysis arxiv cs.lg data ensemble error expert experts fusion methodology moe predictions study through type