Thermometer: Towards Universal Calibration for Large Language Models

arXiv:2403.08819v1 Announce Type: new
Abstract: We consider the issue of calibration in large language models (LLM). Recent studies have found that common interventions such as instruction tuning often result in poorly calibrated LLMs. Although calibration is well-explored in traditional applications, calibrating LLMs is uniquely challenging. These challenges stem as much from the severe computational requirements of LLMs as from their versatility, which allows them to be applied to diverse tasks. Addressing these challenges, we propose THERMOMETER, a calibration approach tailored …

abstract applications arxiv challenges cs.cl cs.lg found issue language language models large language large language models llm llms stat.ml stem studies type universal