Have Seen Me Before? Automating Dataset Updates Towards Reliable and Timely Evaluation

arXiv:2402.11894v1 Announce Type: new
Abstract: Due to the expanding capabilities and pre-training data, Large Language Models (LLMs) are facing increasingly serious evaluation challenges. On one hand, the data leakage issue cause over-estimation on existing benchmarks. On the other hand, periodically curating datasets manually is costly. In this paper, we propose to automate dataset updates for reliable and timely evaluation. The basic idea is to generate unseen and high-quality testing samples based on existing ones to mitigate leakage issues. In specific, …

abstract arxiv benchmarks capabilities challenges cs.cl data data leakage dataset datasets evaluation issue language language models large language large language models llms paper pre-training training training data type updates