Representations learnt by SGD and Adaptive learning rules: Conditions that vary sparsity and selectivity in neural network

arXiv:2201.11653v2 Announce Type: replace
Abstract: From the point of view of the human brain, continual learning can perform various tasks without mutual interference. An effective way to reduce mutual interference can be found in sparsity and selectivity of neurons. According to Aljundi et al. and Hadsell et al., imposing sparsity at the representational level is advantageous for continual learning because sparse neuronal activations encourage less overlap between parameters, resulting in less interference. Similarly, highly selective neural networks are likely to …

abstract arxiv brain continual cs.lg found human interference network neural network neurons reduce rules sparsity tasks type view