Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization

arXiv:2311.06243v2 Announce Type: replace
Abstract: Large foundation models are becoming ubiquitous, but training them from scratch is prohibitively expensive. Thus, efficiently adapting these powerful models to downstream tasks is increasingly important. In this paper, we study a principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream task adaptation. Despite demonstrating good generalizability, OFT still uses a fairly large number of trainable parameters due to the high dimensionality of orthogonal matrices. To address this, we start by examining OFT from …

abstract arxiv cs.ai cs.cl cs.cv cs.lg factorization finetuning foundation good paper paradigm scratch study tasks them training type via