Hard ASH: Sparsity and the right optimizer make a continual learner

arXiv:2404.17651v1 Announce Type: new
Abstract: In class incremental learning, neural networks typically suffer from catastrophic forgetting. We show that an MLP featuring a sparse activation function and an adaptive learning rate optimizer can compete with established regularization techniques in the Split-MNIST task. We highlight the effectiveness of the Adaptive SwisH (ASH) activation function in this context and introduce a novel variant, Hard Adaptive SwisH (Hard ASH) to further enhance the learning retention.

abstract arxiv catastrophic forgetting class continual cs.cv cs.lg function highlight incremental mlp mnist networks neural networks rate regularization show sparsity split type