SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models

arXiv:2403.02509v1 Announce Type: new
Abstract: In recent years, large language models (LLMs) have become increasingly prevalent, offering remarkable text generation capabilities. However, a pressing challenge is their tendency to make confidently wrong predictions, highlighting the critical need for uncertainty quantification (UQ) in LLMs. While previous works have mainly focused on addressing aleatoric uncertainty, the full spectrum of uncertainties, including epistemic, remains inadequately explored. Motivated by this gap, we introduce a novel UQ method, sampling with perturbation for UQ (SPUQ), designed …

abstract arxiv become capabilities challenge cs.ai cs.cl highlighting language language models large language large language models llms predictions quantification text text generation type uncertainty