Prometheus: Inducing Fine-grained Evaluation Capability in Language Models

arXiv:2310.08491v2 Announce Type: replace-cross
Abstract: Recently, using a powerful proprietary Large Language Model (LLM) (e.g., GPT-4) as an evaluator for long-form responses has become the de facto standard. However, for practitioners with large-scale evaluation tasks and custom criteria in consideration (e.g., child-readability), using proprietary LLMs as an evaluator is unreliable due to the closed-source nature, uncontrolled versioning, and prohibitive costs. In this work, we propose Prometheus, a fully open-source LLM that is on par with GPT-4's evaluation capabilities when the …

arxiv capability cs.cl cs.lg evaluation fine-grained language language models prometheus type