Metric-aware LLM inference

arXiv:2403.04182v1 Announce Type: new
Abstract: Large language models (LLMs) have demonstrated strong results on a range of NLP tasks. Typically, outputs are obtained via autoregressive sampling from the LLM's underlying distribution. We show that this inference strategy can be suboptimal for a range of tasks and associated evaluation metrics. As a remedy, we propose metric aware LLM inference: a decision theoretic approach optimizing for custom metrics at inference time. We report improvements over baselines on academic benchmarks and publicly available …

abstract arxiv cs.ai cs.cl distribution evaluation evaluation metrics inference language language models large language large language models llm llms metrics nlp results sampling show strategy tasks type via