BabyBear: Cheap inference triage for expensive language models. (arXiv:2205.11747v1 [cs.CL])

Transformer language models provide superior accuracy over previous models
but they are computationally and environmentally expensive. Borrowing the
concept of model cascading from computer vision, we introduce BabyBear, a
framework for cascading models for natural language processing (NLP) tasks to
minimize cost. The core strategy is inference triage, exiting early when the
least expensive model in the cascade achieves a sufficiently high-confidence
prediction. We test BabyBear on several open source data sets related to
document classification and entity recognition. We …

arxiv inference language language models