An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing

arXiv:2403.16854v1 Announce Type: new
Abstract: We present Expert-Token-Routing, a unified generalist framework that facilitates seamless integration of multiple expert LLMs. Our framework represents expert LLMs as special expert tokens within the vocabulary of a meta LLM. The meta LLM can route to an expert LLM like generating new tokens. Expert-Token-Routing not only supports learning the implicit expertise of expert LLMs from existing instruction dataset but also allows for dynamic extension of new expert LLMs in a plug-and-play manner. It also …

abstract arxiv cs.ai cs.cl expert framework integration llm llms meta meta llm multiple route routing seamless integration token tokens type via