A Study on the Calibration of In-context Learning

arXiv:2312.04021v4 Announce Type: replace-cross
Abstract: Accurate uncertainty quantification is crucial for the safe deployment of machine learning models, and prior research has demonstrated improvements in the calibration of modern language models (LMs). We study in-context learning (ICL), a prevalent method for adapting static LMs through tailored prompts, and examine the balance between performance and calibration across a broad spectrum of natural language understanding and reasoning tasks. Through comprehensive experiments, we observe that, with an increasing number of ICL examples, models …

abstract arxiv balance context cs.ai cs.cl cs.lg deployment improvements in-context learning language language models lms machine machine learning machine learning models modern prior prompts quantification research study through type uncertainty