Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing

arXiv:2404.14618v1 Announce Type: new
Abstract: Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size, while smaller models that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in terms of response quality. Therefore in this work we propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. Our approach uses a router that assigns queries to the small …

abstract arxiv cloud cost cs.ai cs.cl cs.lg deployment devices edge excel hybrid language language models large language large language models llm llms nlp quality query routing servers tasks terms type work