Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level

arXiv:2403.04690v1 Announce Type: cross
Abstract: Neighborhood attention reduces the cost of self attention by restricting each token's attention span to its nearest neighbors. This restriction, parameterized by a window size and dilation factor, draws a spectrum of possible attention patterns between linear projection and self attention. Neighborhood attention, and more generally sliding window attention patterns, have long been bounded by infrastructure, particularly in higher-rank spaces (2-D and 3-D), calling for the development of custom kernels, which have been limited in …

abstract arxiv attention cost cs.ai cs.cv cs.lg faster linear neighbors patterns projection spectrum token type