Stochastic Spiking Attention: Accelerating Attention with Stochastic Computing in Spiking Networks

arXiv:2402.09109v1 Announce Type: cross
Abstract: Spiking Neural Networks (SNNs) have been recently integrated into Transformer architectures due to their potential to reduce computational demands and to improve power efficiency. Yet, the implementation of the attention mechanism using spiking signals on general-purpose computing platforms remains inefficient. In this paper, we propose a novel framework leveraging stochastic computing (SC) to effectively execute the dot-product attention for SNN-based Transformers. We demonstrate that our approach can achieve high classification accuracy ($83.53\%$) on CIFAR-10 within …

abstract architectures arxiv attention computational computing cs.ai cs.ar cs.lg cs.ne eess.sp efficiency general implementation networks neural networks paper platforms power reduce spiking neural networks stochastic transformer type