CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks

arXiv:2404.00566v1 Announce Type: cross
Abstract: To facilitate evaluation of code generation systems across diverse scenarios, we present CodeBenchGen, a framework to create scalable execution-based benchmarks that only requires light guidance from humans. Specifically, we leverage a large language model (LLM) to convert an arbitrary piece of code into an evaluation example, including test cases for execution-based evaluation. We illustrate the usefulness of our framework by creating a dataset, Exec-CSN, which includes 1,931 examples involving 293 libraries revised from code in …

abstract arxiv benchmarks code code generation cs.cl cs.se diverse evaluation example framework guidance humans language language model large language large language model light llm scalable systems test type