Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Texts

arXiv:2403.12918v1 Announce Type: cross
Abstract: Pretrained Language Models (PLMs) have advanced Natural Language Processing (NLP) tasks significantly, but finetuning PLMs on low-resource datasets poses significant challenges such as instability and overfitting. Previous methods tackle these issues by finetuning a strategically chosen subnetwork on a downstream task, while keeping the remaining weights fixed to the pretrained weights. However, they rely on a suboptimal criteria for sub-network selection, leading to suboptimal solutions. To address these limitations, we propose a regularization method based …

abstract advanced arxiv challenges cs.ai cs.cl cs.lg datasets finetuning language language models language processing low natural natural language natural language processing nlp overfitting processing tasks type