Vulnerability Detection with Code Language Models: How Far Are We?

arXiv:2403.18624v1 Announce Type: cross
Abstract: In the context of the rising interest in code language models (code LMs) and vulnerability detection, we study the effectiveness of code LMs for detecting vulnerabilities. Our analysis reveals significant shortcomings in existing vulnerability datasets, including poor data quality, low label accuracy, and high duplication rates, leading to unreliable model performance in realistic vulnerability detection scenarios. Additionally, the evaluation methods used with these datasets are not representative of real-world vulnerability detection.
To address these challenges, …

abstract accuracy analysis arxiv code context cs.cl cs.se data data quality datasets detection language language models lms low quality study type vulnerabilities vulnerability