Softmax Probabilities (Mostly) Predict Large Language Model Correctness on Multiple-Choice Q&A

arXiv:2402.13213v1 Announce Type: cross
Abstract: Although large language models (LLMs) perform impressively on many tasks, overconfidence remains a problem. We hypothesized that on multiple-choice Q&A tasks, wrong answers would be associated with smaller maximum softmax probabilities (MSPs) compared to correct answers. We comprehensively evaluate this hypothesis on ten open-source LLMs and five datasets, and find strong evidence for our hypothesis among models which perform well on the original Q&A task. For the six LLMs with the best Q&A performance, the …

abstract arxiv cs.ai cs.cl cs.lg hypothesis language language model language models large language large language model large language models llms multiple softmax tasks type