[R] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Paper : [\[2403.03507\] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (arxiv.org)](https://arxiv.org/abs/2403.03507)

Codebase: [GaLore (github.com)](https://github.com/jiaweizzhao/GaLore)

Training Large Language Models (LLMs) presents significant memory challenges, predominantly due to the growing size of weights and optimizer states. Common memory-reduction approaches, such as low-rank adaptation (LoRA), add a trainable low-rank matrix to the frozen pre-trained weight in each layer, reducing trainable parameters and optimizer states. However, such approaches typically underperform training with full-rank weights in both pre-training and fine-tuning stages since they limit …

challenges however language language models large language large language models layer llms lora low low-rank adaptation machinelearning matrix memory parameters pre-training training