Parameter-Efficient Fine-Tuning With Adapters

arXiv:2405.05493v1 Announce Type: new
Abstract: In the arena of language model fine-tuning, the traditional approaches, such as Domain-Adaptive Pretraining (DAPT) and Task-Adaptive Pretraining (TAPT), although effective, but computational intensive. This research introduces a novel adaptation method utilizing the UniPELT framework as a base and added a PromptTuning Layer, which significantly reduces the number of trainable parameters while maintaining competitive performance across various benchmarks. Our method employs adapters, which enable efficient transfer of pretrained models to new tasks with minimal retraining …

abstract arena arxiv computational cs.ai cs.cl domain fine-tuning framework language language model layer model fine-tuning novel parameters pretraining research type