Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approach. (arXiv:2207.06267v1 [cs.LG])

Continual learning (CL) over non-stationary data streams remains one of the
long-standing challenges in deep neural networks (DNNs) as they are prone to
catastrophic forgetting. CL models can benefit from self-supervised
pre-training as it enables learning more generalizable task-agnostic features.
However, the effect of self-supervised pre-training diminishes as the length of
task sequences increases. Furthermore, the domain shift between pre-training
data distribution and the task distribution reduces the generalizability of the
learned representations. To address these limitations, we propose Task …

arxiv consolidation continual learning lg representation