NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts

arXiv:2405.04520v1 Announce Type: cross
Abstract: Large language models (LLMs) have manifested strong ability to generate codes for productive activities. However, current benchmarks for code synthesis, such as HumanEval, MBPP, and DS-1000, are predominantly oriented towards introductory tasks on algorithm and data science, insufficiently satisfying challenging requirements prevalent in real-world coding. To fill this gap, we propose NaturalCodeBench (NCB), a challenging code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks. NCB comprises 402 high-quality problems …

arxiv coding cs.cl cs.lg cs.se humaneval natural performance prompts type