SPAFIT: Stratified Progressive Adaptation Fine-tuning for Pre-trained Large Language Models

arXiv:2405.00201v1 Announce Type: new
Abstract: Full fine-tuning is a popular approach to adapt Transformer-based pre-trained large language models to a specific downstream task. However, the substantial requirements for computational power and storage have discouraged its widespread use. Moreover, increasing evidence of catastrophic forgetting and overparameterization in the Transformer architecture has motivated researchers to seek more efficient fine-tuning (PEFT) methods. Commonly known parameter-efficient fine-tuning methods like LoRA and BitFit are typically applied across all layers of the model. We propose a …

abstract adapt architecture arxiv catastrophic forgetting computational cs.ai cs.cl evidence fine-tuning however language language models large language large language models popular power requirements storage transformer transformer architecture type