LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report

arXiv:2405.00732v1 Announce Type: new
Abstract: Low Rank Adaptation (LoRA) has emerged as one of the most widely adopted methods for Parameter Efficient Fine-Tuning (PEFT) of Large Language Models (LLMs). LoRA reduces the number of trainable parameters and memory usage while achieving comparable performance to full fine-tuning. We aim to assess the viability of training and serving LLMs fine-tuned with LoRA in real-world applications. First, we measure the quality of LLMs fine-tuned with quantized low rank adapters across 10 base models …

abstract aim arxiv cs.ai cs.cl cs.lg fine-tuning gpt gpt-4 language language models large language large language models llms lora lora land low memory parameters peft performance report technical type usage while