SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models

arXiv:2401.08295v2 Announce Type: replace
Abstract: The continual learning (CL) ability is vital for deploying large language models (LLMs) in the dynamic world. Existing methods devise the learning module to acquire task-specific knowledge with parameter-efficient tuning (PET) block and the selection module to pick out the corresponding one for the testing input, aiming at handling the challenges of catastrophic forgetting and knowledge transfer in CL. However, these methods tend to address only one of the challenges, ignoring the potential of aligning …

abstract arxiv attention block continual cs.cl dynamic framework knowledge language language models large language large language models llms pet type vital world