CodeEditorBench: Evaluating Code Editing Capability of Large Language Models

arXiv:2404.03543v1 Announce Type: cross
Abstract: Large Language Models (LLMs) for code are rapidly evolving, with code editing emerging as a critical capability. We introduce CodeEditorBench, an evaluation framework designed to rigorously assess the performance of LLMs in code editing tasks, including debugging, translating, polishing, and requirement switching. Unlike existing benchmarks focusing solely on code generation, CodeEditorBench emphasizes real-world scenarios and practical aspects of software development. We curate diverse coding challenges and scenarios from five sources, covering various programming languages, complexity …

abstract arxiv benchmarks capability code cs.ai cs.cl cs.lg cs.se debugging editing evaluation framework language language models large language large language models llms performance tasks type