Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgetting

arXiv:2304.00933v3 Announce Type: replace
Abstract: Continual learning research has shown that neural networks suffer from catastrophic forgetting "at the output level", but it is debated whether this is also the case at the level of learned representations. Multiple recent studies ascribe representations a certain level of innate robustness against forgetting - that they only forget minimally and no critical information. We revisit and expand upon the experiments that revealed this difference in forgetting and illustrate the coexistence of two phenomena …

abstract arxiv case catastrophic forgetting continual cs.cv cs.lg feature issue knowledge multiple networks neural networks research robustness studies type