Spikformer: When Spiking Neural Network Meets Transformer. (arXiv:2209.15425v1 [cs.NE])

We consider two biologically plausible structures, the Spiking Neural Network
(SNN) and the self-attention mechanism. The former offers an energy-efficient
and event-driven paradigm for deep learning, while the latter has the ability
to capture feature dependencies, enabling Transformer to achieve good
performance. It is intuitively promising to explore the marriage between them.
In this paper, we consider leveraging both self-attention capability and
biological properties of SNNs, and propose a novel Spiking Self Attention (SSA)
as well as a powerful framework, …

arxiv network neural network spiking neural network transformer