What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization. (arXiv:2305.07615v1 [cs.CL])

Summarization models often generate text that is poorly calibrated to quality
metrics because they are trained to maximize the likelihood of a single
reference (MLE). To address this, recent work has added a calibration step,
which exposes a model to its own ranked outputs to improve relevance or, in a
separate line of work, contrasts positive and negative sets to improve
faithfulness. While effective, much of this work has focused on how to generate
and optimize these sets. Less is …

arxiv likelihood metrics mle quality reference summarization text work