NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness

Existing evaluation benchmarks of language models of code (code LMs) focus almost exclusively on whether the LMs can generate functionally-correct code. In real-world software engineering, developers think beyond functional correctness. They have requirements on "how" a functionality should be implemented to meet overall system design objectives like efficiency, security, and maintainability. They would also trust the code LMs more if the LMs demonstrate robust understanding of requirements and code semantics.
We propose a new benchmark NoFunEval to evaluate code LMs …

benchmarks beyond code cs.ai cs.cl cs.lg cs.se design developers efficiency engineering evaluation focus functional funny generate language language models lms requirements security software software engineering think world