Hydra Attention: Efficient Attention with Many Heads. (arXiv:2209.07484v1 [cs.CV])

While transformers have begun to dominate many tasks in vision, applying them
to large images is still computationally difficult. A large reason for this is
that self-attention scales quadratically with the number of tokens, which in
turn, scales quadratically with the image size. On larger images (e.g., 1080p),
over 60% of the total computation in the network is spent solely on creating
and applying attention matrices. We take a step toward solving this issue by
introducing Hydra Attention, an extremely …

arxiv attention hydra