Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selection

arXiv:2405.02134v1 Announce Type: new
Abstract: Researchers and practitioners operating on a limited budget face the cost-performance trade-off dilemma. The challenging decision often centers on whether to use a large LLM with better performance or a smaller one with reduced costs. This has motivated recent research in the optimisation of LLM calls. Either a cascading strategy is used, where a smaller LLM or both are called sequentially, or a routing strategy is used, where only one model is ever called. Both …

abstract arxiv budget cost costs cs.cl decision face language language models large language large language models llm optimisation performance research researchers trade trade-off type uncertainty