The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?

arXiv:2402.19475v1 Announce Type: cross
Abstract: While language models are increasingly more proficient at code generation, they still frequently generate incorrect programs. Many of these programs are obviously wrong, but others are more subtle and pass weaker correctness checks such as being able to compile. In this work, we focus on these counterfeit samples: programs sampled from a language model that 1) have a high enough log-probability to be generated at a moderate temperature and 2) pass weak correctness checks. Overall, …

abstract arxiv checks code code generation counterfeit cs.ai cs.lg cs.se generate language language models type