LayerNorm: A key component in parameter-efficient fine-tuning

arXiv:2403.20284v1 Announce Type: cross
Abstract: Fine-tuning a pre-trained model, such as Bidirectional Encoder Representations from Transformers (BERT), has been proven to be an effective method for solving many natural language processing (NLP) tasks. However, due to the large number of parameters in many state-of-the-art NLP models, including BERT, the process of fine-tuning is computationally expensive. One attractive solution to this issue is parameter-efficient fine-tuning, which involves modifying only a minimal segment of the model while keeping the remainder unchanged. Yet, …

abstract art arxiv bert cs.cl cs.lg encoder fine-tuning however key language language processing natural natural language natural language processing nlp nlp models parameters pre-trained model process processing state tasks transformers type