Language Model Cascades: Token-level uncertainty and beyond

arXiv:2404.10136v1 Announce Type: cross
Abstract: Recent advances in language models (LMs) have led to significant improvements in quality on complex NLP tasks, but at the expense of increased inference costs. Cascading offers a simple strategy to achieve more favorable cost-quality tradeoffs: here, a small model is invoked for most "easy" instances, while a few "hard" instances are deferred to the large model. While the principles underpinning cascading are well-studied for classification tasks - with deferral based on predicted class uncertainty …

abstract advances arxiv beyond cost costs cs.ai cs.cl cs.lg easy improvements inference inference costs instances language language model language models lms nlp quality simple small strategy tasks token type uncertainty