GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

arXiv:2403.03507v1 Announce Type: new
Abstract: Training Large Language Models (LLMs) presents significant memory challenges, predominantly due to the growing size of weights and optimizer states. Common memory-reduction approaches, such as low-rank adaptation (LoRA), add a trainable low-rank matrix to the frozen pre-trained weight in each layer, reducing trainable parameters and optimizer states. However, such approaches typically underperform training with full-rank weights in both pre-training and fine-tuning stages since they limit the parameter search to a low-rank subspace and alter the …

abstract arxiv challenges cs.lg gradient language language models large language large language models layer llm llms lora low low-rank adaptation matrix memory parameters projection training type