Bridging the Gap between Different Vocabularies for LLM Ensemble

arXiv:2404.09492v1 Announce Type: new
Abstract: Ensembling different large language models (LLMs) to unleash their complementary potential and harness their individual strengths is highly valuable. Nevertheless, vocabulary discrepancies among various LLMs have constrained previous studies to either selecting or blending completely generated outputs. This limitation hinders the dynamic correction and enhancement of outputs during the generation process, resulting in a limited capacity for effective ensemble. To address this issue, we propose a novel method to Ensemble LLMs via Vocabulary Alignment (EVA). …

abstract arxiv cs.cl dynamic ensemble gap generated harness language language models large language large language models llm llms studies type