Evaluating Large Language Models with Runtime Behavior of Program Execution

arXiv:2403.16437v1 Announce Type: cross
Abstract: Large language models for code (i.e., code LLMs) have shown strong code understanding and generation capabilities. To evaluate the capabilities of code LLMs in various aspects, many benchmarks have been proposed (e.g., HumanEval and ClassEval). Code reasoning is one of the most essential abilities of code LLMs, but existing benchmarks for code reasoning are not sufficient. Typically, they focus on predicting the input and output of a program, ignoring the evaluation of the intermediate behavior …

abstract arxiv behavior benchmarks capabilities code code llms code understanding cs.cl cs.se humaneval language language models large language large language models llms reasoning type understanding