Structure-aware Fine-tuning for Code Pre-trained Models

arXiv:2404.07471v1 Announce Type: cross
Abstract: Over the past few years, we have witnessed remarkable advancements in Code Pre-trained Models (CodePTMs). These models achieved excellent representation capabilities by designing structure-based pre-training tasks for code. However, how to enhance the absorption of structural knowledge when fine-tuning CodePTMs still remains a significant challenge. To fill this gap, in this paper, we present Structure-aware Fine-tuning (SAT), a novel structure-enhanced and plug-and-play fine-tuning method for CodePTMs. We first propose a structure loss to quantify the …

abstract arxiv capabilities challenge code cs.ai cs.cl cs.se designing fine-tuning gap however knowledge pre-trained models pre-training representation tasks training type