Self-Consistency Boosts Calibration for Math Reasoning

arXiv:2403.09849v1 Announce Type: new
Abstract: Calibration, which establishes the correlation between accuracy and model confidence, is important for LLM development. We design three off-the-shelf calibration methods based on self-consistency (Wang et al., 2022) for math reasoning tasks. Evaluation on two popular benchmarks (GSM8K and MathQA) using strong open-source LLMs (Mistral and LLaMA2), our methods better bridge model confidence and accuracy than existing methods based on p(True) (Kadavath et al., 2022) or logit (Kadavath et al., 2022).

abstract accuracy arxiv benchmarks confidence correlation cs.ai cs.cl design development evaluation llama2 llm llm development llms math mistral popular reasoning tasks type