Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation

arXiv:2404.04316v1 Announce Type: new
Abstract: With the increasingly powerful performances and enormous scales of Pretrained Language Models (PLMs), promoting parameter efficiency in fine-tuning has become a crucial need for effective and efficient adaptation to various downstream tasks. One representative line of fine-tuning methods is Orthogonal Fine-tuning (OFT), which rigorously preserves the angular distances within the parameter space to preserve the pretrained knowledge. Despite the empirical effectiveness, OFT still suffers low parameter efficiency at $\mathcal{O}(d^2)$ and limited capability of downstream adaptation. …

abstract angular arxiv become cs.ai cs.cl cs.lg efficiency fine-tuning language language models line performances rotation tasks type via