Activation Steering for Robust Type Prediction in CodeLLMs

arXiv:2404.01903v1 Announce Type: cross
Abstract: Contemporary LLMs pretrained on code are capable of succeeding at a wide variety of programming tasks. However, their performance is very sensitive to syntactic features, such as the names of variables and types, the structure of code, and presence of type hints. We contribute an inference-time technique to make CodeLLMs more robust to syntactic distractors that are semantically irrelevant. Our methodology relies on activation steering, which involves editing internal model activations to steer the model …

abstract arxiv code cs.cl cs.lg cs.pl features however inference llms performance prediction programming robust tasks type types variables