WDMoE: Wireless Distributed Large Language Models with Mixture of Experts

arXiv:2405.03131v1 Announce Type: cross
Abstract: Large Language Models (LLMs) have achieved significant success in various natural language processing tasks, but how wireless communications can support LLMs has not been extensively studied. In this paper, we propose a wireless distributed LLMs paradigm based on Mixture of Experts (MoE), named WDMoE, deploying LLMs collaboratively across edge servers of base station (BS) and mobile devices in the wireless communications system. Specifically, we decompose the MoE layer in LLMs by deploying the gating network …

abstract arxiv communications cs.ai cs.it cs.lg distributed experts language language models language processing large language large language models llms math.it mixture of experts moe natural natural language natural language processing paper paradigm processing success support tasks type wireless wireless communications