When to Trust LLMs: Aligning Confidence with Response Quality

arXiv:2404.17287v1 Announce Type: new
Abstract: Despite the success of large language models (LLMs) in natural language generation, much evidence shows that LLMs may produce incorrect or nonsensical text. This limitation highlights the importance of discerning when to trust LLMs, especially in safety-critical domains. Existing methods, which rely on verbalizing confidence to tell the reliability by inducing top-k responses and sampling-aggregating multiple responses, often fail, due to the lack of objective guidance of confidence. To address this, we propose CONfidence-Quality-ORDerpreserving alignment …

abstract arxiv confidence cs.cl domains evidence highlights importance language language generation language models large language large language models llms natural natural language natural language generation quality safety safety-critical shows success text trust type