Evaluating Large Language Models at Evaluating Instruction Following

arXiv:2310.07641v2 Announce Type: replace-cross
Abstract: As research in large language models (LLMs) continues to accelerate, LLM-based evaluation has emerged as a scalable and cost-effective alternative to human evaluations for comparing the ever increasing list of models. This paper investigates the efficacy of these ``LLM evaluators'', particularly in using them to assess instruction following, a metric that gauges how closely generated text adheres to the given instruction. We introduce a challenging meta-evaluation benchmark, LLMBar, designed to test the ability of an …

abstract arxiv cost cs.cl cs.lg evaluation ever human language language models large language large language models list llm llms paper research scalable them type