Faster Attention Is What You Need: A Fast Self-Attention Neural Network Backbone Architecture for the Edge via Double-Condensing Attention Condensers. (arXiv:2208.06980v1 [cs.CV])

With the growing adoption of deep learning for on-device TinyML applications,
there has been an ever-increasing demand for more efficient neural network
backbones optimized for the edge. Recently, the introduction of attention
condenser networks have resulted in low-footprint, highly-efficient,
self-attention neural networks that strike a strong balance between accuracy
and speed. In this study, we introduce a new faster attention condenser design
called double-condensing attention condensers that enable more condensed
feature embedding. We further employ a machine-driven design exploration
strategy …

architecture arxiv attention cv edge network neural network self-attention the edge