SCT: A Simple Baseline for Parameter-Efficient Fine-Tuning via Salient Channels

arXiv:2309.08513v3 Announce Type: replace
Abstract: Pre-trained vision transformers have strong representation benefits to various downstream tasks. Recently, many parameter-efficient fine-tuning (PEFT) methods have been proposed, and their experiments demonstrate that tuning only 1% of extra parameters could surpass full fine-tuning in low-data resource scenarios. However, these methods overlook the task-specific information when fine-tuning diverse downstream tasks. In this paper, we propose a simple yet effective method called "Salient Channel Tuning" (SCT) to leverage the task-specific information by forwarding the model …

abstract arxiv benefits channels cs.ai cs.cv data extra fine-tuning however low parameters peft representation simple tasks transformers type via vision vision transformers