LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

arXiv:2403.07974v1 Announce Type: cross
Abstract: Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry. However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEval, MBPP) are no longer sufficient for assessing their capabilities. In this work, we propose LiveCodeBench, a comprehensive and contamination-free evaluation of LLMs for code, which continuously collects new problems over time from contests across three competition platforms, namely LeetCode, AtCoder, …

abstract academia applications arxiv benchmarks code cs.cl cs.lg cs.se evaluation free however humaneval industry language language models large language large language models llms type