Are LLM-based Evaluators Confusing NLG Quality Criteria?

arXiv:2402.12055v1 Announce Type: new
Abstract: Some prior work has shown that LLMs perform well in NLG evaluation for different tasks. However, we discover that LLMs seem to confuse different evaluation criteria, which reduces their reliability. For further verification, we first consider avoiding issues of inconsistent conceptualization and vague expression in existing NLG quality criteria themselves. So we summarize a clear hierarchical classification system for 11 common aspects with corresponding different criteria from previous studies involved. Inspired by behavioral testing, we …

abstract arxiv cs.cl evaluation llm llms nlg prior quality reliability tasks type verification work