On the Relationship between Skill Neurons and Robustness in Prompt Tuning

arXiv:2309.12263v2 Announce Type: replace
Abstract: Prompt Tuning is a popular parameter-efficient finetuning method for pre-trained large language models (PLMs). Based on experiments with RoBERTa, it has been suggested that Prompt Tuning activates specific neurons in the transformer's feed-forward networks, that are highly predictive and selective for the given task. In this paper, we study the robustness of Prompt Tuning in relation to these "skill neurons", using RoBERTa and T5. We show that prompts tuned for a specific task are transferable …

abstract arxiv cs.cl finetuning language language models large language large language models networks neurons popular predictive prompt prompt tuning relationship roberta robustness transformer type