Language Models for Code Completion: A Practical Evaluation

arXiv:2402.16197v1 Announce Type: cross
Abstract: Transformer-based language models for automatic code completion have shown great promise so far, yet the evaluation of these models rarely uses real data. This study provides both quantitative and qualitative assessments of three public code language models when completing real-world code. We first developed an open-source IDE extension, Code4Me, for the online evaluation of the models. We collected real auto-completion usage data for over a year from more than 1200 users, resulting in over 600K …

abstract arxiv code code completion cs.lg cs.pl cs.se data evaluation ide language language models practical public quantitative real data study transformer type world