Tied-Lora: Enhancing parameter efficiency of LoRA with weight tying

arXiv:2311.09578v2 Announce Type: replace-cross
Abstract: We introduce Tied-LoRA, a novel paradigm leveraging weight tying and selective training to enhance the parameter efficiency of Low-rank Adaptation (LoRA). Our exploration encompasses different plausible combinations of parameter training and freezing, coupled with weight tying, aimed at identifying the optimal trade-off between performance and the count of trainable parameters. Across $5$ diverse tasks and two foundational language models with different parameter counts, our experiments provide comprehensive insights into the inherent trade-offs between efficiency and …

abstract arxiv cs.ai cs.cl cs.lg efficiency exploration lora low low-rank adaptation novel paradigm performance trade trade-off training type