Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

arXiv:2306.13063v2 Announce Type: replace
Abstract: Empowering large language models to accurately express confidence in their answers is essential for trustworthy decision-making. Previous confidence elicitation methods, which primarily rely on white-box access to internal model information or model fine-tuning, have become less suitable for LLMs, especially closed-source commercial APIs. This leads to a growing need to explore the untapped area of black-box approaches for LLM uncertainty estimation. To better break down the problem, we define a systematic framework with three components: …

abstract arxiv become box commercial confidence cs.cl decision evaluation express fine-tuning information language language models large language large language models llms making model fine-tuning trustworthy type uncertainty