ML-Bench: Evaluating Large Language Models for Code Generation in Repository-Level Machine Learning Tasks

arXiv:2311.09835v2 Announce Type: replace
Abstract: While Large Language Models (LLMs) have demonstrated proficiency in code generation benchmarks, translating these results into practical development scenarios - where leveraging existing repository-level libraries is the norm - remains challenging. To bridge the gap between lab-scale benchmarks and real-world coding practices, we introduce ML-Bench: a novel benchmark designed to assess LLMs' ability to integrate and utilize repository-level open-source libraries to complete machine learning tasks. ML-Bench comprises a diverse set of 9,641 samples across 169 …

arxiv code code generation cs.ai cs.cl language language models large language large language models machine machine learning tasks type