Tensor Decomposition Based Attention Module for Spiking Neural Networks

arXiv:2310.14576v2 Announce Type: replace
Abstract: The attention mechanism has been proven to be an effective way to improve spiking neural network (SNN). However, based on the fact that the current SNN input data flow is split into tensors to process on GPUs, none of the previous works consider the properties of tensors to implement an attention module. This inspires us to rethink current SNN from the perspective of tensor-relevant theories. Using tensor decomposition, we design the \textit{projected full attention} (PFA) …

abstract arxiv attention cs.cv cs.lg current data data flow flow gpus however network networks neural network neural networks process snn spiking neural network spiking neural networks split tensor type