Ripple Attention for Visual Perception with Sub-quadratic Complexity. (arXiv:2110.02453v2 [cs.CV] UPDATED)

Transformer architectures are now central to sequence modeling tasks. At its
heart is the attention mechanism, which enables effective modeling of long-term
dependencies in a sequence. Recently, transformers have been successfully
applied in the computer vision domain, where 2D images are first segmented into
patches and then treated as 1D sequences. Such linearization, however, impairs
the notion of spatial locality in images, which bears important visual clues.
To bridge the gap, we propose ripple attention, a sub-quadratic attention
mechanism for …

arxiv attention complexity cv perception ripple