Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activations. (arXiv:2203.06514v2 [cs.LG] UPDATED)

Continual/lifelong learning from a non-stationary input data stream is a
cornerstone of intelligence. Despite their phenomenal performance in a wide
variety of applications, deep neural networks are prone to forgetting their
previously learned information upon learning new ones. This phenomenon is
called "catastrophic forgetting" and is deeply rooted in the
stability-plasticity dilemma. Overcoming catastrophic forgetting in deep neural
networks has become an active field of research in recent years. In particular,
gradient projection-based methods have recently shown exceptional performance
at …

arxiv continual dropout learning lg null space sparsity