Redundancy and Concept Analysis for Code-trained Language Models

arXiv:2305.00875v2 Announce Type: replace-cross
Abstract: Code-trained language models have proven to be highly effective for various code intelligence tasks. However, they can be challenging to train and deploy for many software engineering applications due to computational bottlenecks and memory constraints. Implementing effective strategies to address these issues requires a better understanding of these 'black box' models. In this paper, we perform the first neuron-level analysis for source code models to identify \textit{important} neurons within latent representations. We achieve this by …

abstract analysis applications arxiv bottlenecks code code intelligence computational concept constraints cs.ai cs.lg cs.se deploy engineering intelligence language language models memory redundancy software software engineering strategies tasks train type understanding