OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models

arXiv:2401.06628v2 Announce Type: replace
Abstract: Advancing automated programming necessitates robust and comprehensive code generation benchmarks, yet current evaluation frameworks largely neglect object-oriented programming (OOP) in favor of functional programming (FP), e.g., HumanEval and MBPP. To address this, our study introduces a pioneering OOP-focused benchmark, featuring 431 Python programs that encompass essential OOP concepts and features like classes and encapsulation methods. We propose a novel evaluation metric, pass@o, tailored for OOP, enhancing traditional pass@k measures. Our evaluation of 23 leading large …

arxiv benchmark cs.cl evaluation language language models large language large language models object-oriented oop programming type