Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models

arXiv:2405.00433v1 Announce Type: new
Abstract: Activity and parameter sparsity are two standard methods of making neural networks computationally more efficient. Event-based architectures such as spiking neural networks (SNNs) naturally exhibit activity sparsity, and many methods exist to sparsify their connectivity by pruning weights. While the effect of weight pruning on feed-forward SNNs has been previously studied for computer vision tasks, the effects of pruning for complex sequence tasks like language modeling are less well studied since SNNs have traditionally struggled …

abstract architectures arxiv connectivity cs.ai cs.lg cs.ne event language language models making networks neural networks neuromorphic pruning sparsity spiking neural networks standard type while