Do Large Code Models Understand Programming Concepts? A Black-box Approach

Large Language Models' success on text generation has also made them better at code generation and coding tasks. While a lot of work has demonstrated their remarkable performance on tasks such as code completion and editing, it is still unclear as to why. We help bridge this gap by exploring to what degree auto-regressive models understand the logical constructs of the underlying programs. We propose Counterfactual Analysis for Programming Concept Predicates (CACP) as a counterfactual testing framework to evaluate whether …

box bridge code code completion code generation coding concepts cs.ai cs.lg cs.pl cs.se editing gap language language models large language large language models performance programming success tasks text text generation them work