Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition

arXiv:2404.08008v1 Announce Type: new
Abstract: The past years have witnessed a proliferation of large language models (LLMs). Yet, automated and unbiased evaluation of LLMs is challenging due to the inaccuracy of standard metrics in reflecting human preferences and the inefficiency in sampling informative and diverse test examples. While human evaluation remains the gold standard, it is expensive and time-consuming, especially when dealing with a large number of testing samples. To address this problem, we propose a sample-efficient human evaluation method …

abstract arxiv automated competition cs.cl cs.hc cs.lg diverse evaluation examples human language language models large language large language models llms metrics sample sampling standard test type unbiased via