PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLM

arXiv:2401.03855v2 Announce Type: replace
Abstract: Driven by the surge in code generation using large language models (LLMs), numerous benchmarks have emerged to evaluate these LLMs capabilities. We conducted a large-scale human evaluation of HumanEval and MBPP, two popular benchmarks for Python code generation, analyzing their diversity and difficulty. Our findings unveil a critical bias towards a limited set of programming concepts, neglecting most of the other concepts entirely. Furthermore, we uncover a worrying prevalence of easy tasks, potentially inflating model …

abstract arxiv benchmark benchmarks capabilities code code generation cs.ai cs.cl diversity evaluation human humaneval language language models large language large language models llm llms popular python scale type