Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models

arXiv:2405.01535v1 Announce Type: new
Abstract: Proprietary LMs such as GPT-4 are often employed to assess the quality of responses from various LMs. However, concerns including transparency, controllability, and affordability strongly motivate the development of open-source LMs specialized in evaluations. On the other hand, existing open evaluator LMs exhibit critical shortcomings: 1) they issue scores that significantly diverge from those assigned by humans, and 2) they lack the flexibility to perform both direct assessment and pairwise ranking, the two most prevalent …

arxiv cs.cl language language model language models open source open source language model prometheus type