Analyzing the Performance of Large Language Models on Code Summarization

arXiv:2404.08018v1 Announce Type: cross
Abstract: Large language models (LLMs) such as Llama 2 perform very well on tasks that involve both natural language and source code, particularly code summarization and code generation. We show that for the task of code summarization, the performance of these models on individual examples often depends on the amount of (subword) token overlap between the code and the corresponding reference natural language descriptions in the dataset. This token overlap arises because the reference descriptions in …

abstract arxiv code code generation cs.ai cs.cl cs.se examples language language models large language large language models llama llama 2 llms natural natural language performance show summarization tasks type