LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses

arXiv:2310.19208v2 Announce Type: replace
Abstract: A model is considered well-calibrated when its probability estimate aligns with the actual likelihood of the output being correct. Calibrating language models (LMs) is crucial, as it plays a vital role in detecting and mitigating hallucinations of LMs as well as building more trustworthy models. However, standard calibration techniques may not be suited for LM calibration. For instance, post-processing methods such as temperature scaling do not reorder the candidate generations. On the other hand, training-based …

abstract arxiv building cs.cl form hallucinations language language model language models likelihood lms probability responses role trustworthy type vital