Benchmarking Educational Program Repair

arXiv:2405.05347v1 Announce Type: cross
Abstract: The emergence of large language models (LLMs) has sparked enormous interest due to their potential application across a range of educational tasks. For example, recent work in programming education has used LLMs to generate learning resources, improve error messages, and provide feedback on code. However, one factor that limits progress within the field is that much of the research uses bespoke datasets and different evaluation metrics, making direct comparisons between results unreliable. Thus, there is …

abstract application arxiv benchmarking code cs.ai cs.cl cs.cy cs.se education educational emergence error error messages example feedback generate however language language models large language large language models llms messages programming progress repair resources tasks type work