Fisher Mask Nodes for Language Model Merging

arXiv:2403.09891v1 Announce Type: cross
Abstract: Fine-tuning pre-trained models provides significant advantages in downstream performance. The ubiquitous nature of pre-trained models such as BERT and its derivatives in natural language processing has also led to a proliferation of task-specific fine-tuned models. As these models typically only perform one task well, additional training or ensembling is required in multi-task scenarios. The growing field of model merging provides a solution, dealing with the challenge of combining multiple task-specific models into a single multi-task …

abstract advantages arxiv bert cs.ai cs.cl cs.lg derivatives fine-tuning fisher language language model language processing merging natural natural language natural language processing nature nodes performance pre-trained models processing training type