PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

arXiv:2404.02948v1 Announce Type: new
Abstract: As the parameters of LLMs expand, the computational cost of fine-tuning the entire model becomes prohibitive. To address this challenge, we introduce a PEFT method, Principal Singular values and Singular vectors Adaptation (PiSSA), which optimizes a significantly reduced parameter space while achieving or surpassing the performance of full-parameter fine-tuning. PiSSA is inspired by Intrinsic SAID, which suggests that pre-trained, over-parametrized models inhabit a space of low intrinsic dimension. Consequently, PiSSA represents a matrix W within …

abstract arxiv challenge computational cost cs.ai cs.lg expand fine-tuning language language models large language large language models llms parameters peft singular space type values vectors