Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach

arXiv:2404.15993v1 Announce Type: cross
Abstract: Large language models (LLMs) are highly capable of many tasks but they can sometimes generate unreliable or inaccurate outputs. To tackle this issue, this paper studies the problem of uncertainty estimation and calibration for LLMs. We begin by formulating the uncertainty estimation problem for LLMs and then propose a supervised approach that takes advantage of the labeled datasets and estimates the uncertainty of the LLMs' responses. Based on the formulation, we illustrate the difference between …

abstract arxiv cs.cl cs.lg generate issue language language models large language large language models llms paper quantification simple studies tasks type uncertainty