F-MALLOC: Feed-forward Memory Allocation for Continual Learning in Neural Machine Translation

arXiv:2404.04846v1 Announce Type: new
Abstract: In the evolving landscape of Neural Machine Translation (NMT), the pretrain-then-finetune paradigm has yielded impressive results. However, the persistent challenge of Catastrophic Forgetting (CF) remains a hurdle. While previous work has introduced Continual Learning (CL) methods to address CF, these approaches grapple with the delicate balance between avoiding forgetting and maintaining system extensibility. To address this, we propose a CL method, named $\textbf{F-MALLOC}$ ($\textbf{F}$eed-forward $\textbf{M}$emory $\textbf{ALLOC}ation)$. F-MALLOC is inspired by recent insights highlighting that feed-forward …

abstract arxiv catastrophic forgetting challenge continual cs.cl however landscape machine machine translation memory neural machine translation paradigm results translation type work