Exploration of Parameters-efficient fine tuning methods (LoRA/MoRA/DoRA) in LLM

Exploration of Parameters-efficient fine-tuning methods (LoRA/MoRA/DoRA) in LLM

Introduction

Pre-trained models using extensive general domain datasets have demonstrated impressive generalization abilities, benefiting a wide range of applications, from natural language processing (NLP) to multi-modal tasks. Adapting these general models for specific downstream tasks typically involves full fine-tuning (FT), which retrains all model parameters. However, as models and datasets grow in size, the cost of fine-tuning the entire model becomes very high.

To address this issue, parameter-efficient fine-tuning (PEFT) methods have …

fine-tuning llm lora