From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline

arXiv:2406.11939v1 Announce Type: cross
Abstract: The rapid evolution of language models has necessitated the development of more challenging benchmarks. Current static benchmarks often struggle to consistently distinguish between the capabilities of different models and fail to align with real-world user preferences. On the other hand, live crowd-sourced platforms like the Chatbot Arena collect a wide range of natural prompts and user feedback. However, these prompts vary in sophistication and the feedback cannot be applied offline to new models. In order …

abstract arena arena-hard arxiv benchmarks capabilities cs.ai cs.cl cs.lg current data development evolution fail language language models pipeline platforms quality struggle type world