LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores

arXiv:2311.09766v2 Announce Type: replace
Abstract: Automatic evaluation of generated textual content presents an ongoing challenge within the field of NLP. Given the impressive capabilities of modern language models (LMs) across diverse NLP tasks, there is a growing trend to employ these models in creating innovative evaluation metrics for automated assessment of generation tasks. This paper investigates a pivotal question: Do language model-driven evaluation metrics inherently exhibit bias favoring texts generated by the same underlying language model? Specifically, we assess whether …

abstract arxiv assessment automated capabilities challenge cs.cl diverse evaluation evaluation metrics generated language language models llms lms metrics modern nlp tasks textual trend type