MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMs

arXiv:2402.11756v1 Announce Type: cross
Abstract: Generative Large Language Models (LLMs) are widely utilized for their excellence in various tasks. However, their tendency to produce inaccurate or misleading outputs poses a potential risk, particularly in high-stakes environments. Therefore, estimating the correctness of generative LLM outputs is an important task for enhanced reliability. Uncertainty Estimation (UE) in generative LLMs is an evolving domain, where SOTA probability-based methods commonly employ length-normalized scoring. In this work, we propose Meaning-Aware Response Scoring (MARS) as an …

arxiv cs.cl cs.lg generative llms mars meaning scoring type uncertainty