Hyperparameter Optimization for Large Language Model Instruction-Tuning

The fine-tuning of Large Language Models (LLMs) has enabled them to recently achieve milestones in natural language processing applications. The emergence of ever larger LLMs has paved the way for more efficient fine-tuning methods. Among these, the Low-Rank Adaptation (LoRA) method keeps most of the weights of the pre-trained LLM frozen while introducing a low-rank decomposition of the weight matrix, enabling the tuning of only a very small proportion of the network. The performance on downstream tasks of models fine-tuned …

applications cs.cl emergence fine-tuning hyperparameter language language model language models language processing large language large language model large language models llm llms lora low low-rank adaptation math.oc milestones natural natural language natural language processing optimization processing them