Intelligent Learning Rate Distribution to reduce Catastrophic Forgetting in Transformers

arXiv:2404.01317v1 Announce Type: cross
Abstract: Pretraining language models on large text corpora is a common practice in natural language processing. Fine-tuning of these models is then performed to achieve the best results on a variety of tasks. In this paper, we investigate the problem of catastrophic forgetting in transformer neural networks and question the common practice of fine-tuning with a flat learning rate for the entire network in this context. We perform a hyperparameter optimization process to find learning rate …

abstract arxiv catastrophic forgetting cs.ai cs.cl cs.lg distribution fine-tuning intelligent language language models language processing natural natural language natural language processing paper practice pretraining processing rate reduce results tasks text transformer transformers type