When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method

arXiv:2402.17193v1 Announce Type: cross
Abstract: While large language models (LLMs) often adopt finetuning to unlock their capabilities for downstream applications, our understanding on the inductive biases (especially the scaling properties) of different finetuning methods is still limited. To fill this gap, we conduct systematic experiments studying whether and how different scaling factors, including LLM model size, pretraining data size, new finetuning parameter size and finetuning data size, affect the finetuning performance. We consider two types of finetuning -- full-model tuning …

abstract applications arxiv biases capabilities cs.cl cs.lg data finetuning gap inductive language language models large language large language models llm llms scaling studying type understanding