Scalable Language Model with Generalized Continual Learning

arXiv:2404.07470v1 Announce Type: new
Abstract: Continual learning has gained increasing importance as it facilitates the acquisition and refinement of scalable knowledge and skills in language models. However, existing methods typically encounter strict limitations and challenges in real-world scenarios, such as reliance on experience replay, optimization constraints, and inference task-ID. In this study, we introduce the Scalable Language Model (SLM) to overcome these limitations within a more challenging and generalized setting, representing a significant advancement toward practical applications for continual learning. …

abstract acquisition arxiv challenges constraints continual cs.cl experience generalized however importance inference knowledge language language model language models limitations optimization reliance scalable skills study type world