Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks

arXiv:2404.16966v1 Announce Type: new
Abstract: Benchmarks have emerged as the central approach for evaluating Large Language Models (LLMs). The research community often relies on a model's average performance across the test prompts of a benchmark to evaluate the model's performance. This is consistent with the assumption that the test prompts within a benchmark represent a random sample from a real-world distribution of interest. We note that this is generally not the case; instead, we hold that the distribution of interest …

abstract arxiv assumptions benchmark benchmarks community consistent cs.cl evaluation language language models large language large language models llm llms performance prompts research research community robustness s performance test type