Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models

arXiv:2404.18796v1 Announce Type: new
Abstract: As Large Language Models (LLMs) have become more advanced, they have outpaced our abilities to accurately evaluate their quality. Not only is finding data to adequately probe particular model properties difficult, but evaluating the correctness of a model's freeform generation alone is a challenge. To address this, many evaluations now rely on using LLMs themselves as judges to score the quality of outputs from other LLMs. Evaluations most commonly use a single large model like …

abstract advanced arxiv become cs.ai cs.cl data diverse judges language language models large language large language models llm llms panel probe quality type