Evaluate What You Can't Evaluate: Unassessable Quality for Generated Response

arXiv:2305.14658v3 Announce Type: replace
Abstract: LLMs (large language models) such as ChatGPT have shown remarkable language understanding and generation capabilities. Although reference-free evaluators based on LLMs show better human alignment than traditional reference-based evaluators, there are many challenges in using reference-free evaluators based on LLMs. Reference-free evaluators are more suitable for open-ended examples with different semantics responses. But not all examples are open-ended. For closed-ended examples with unique correct semantic response, reference-free evaluators will still consider it high quality when …

abstract alignment arxiv capabilities challenges chatgpt cs.cl free generated human language language models language understanding large language large language models llms quality reference show type understanding