[R] Hydra Attention: Efficient Attention with Many Heads - Meta AI 2022 - 197x faster than standard attention

Paper: [https://arxiv.org/abs/2209.07484](https://arxiv.org/abs/2209.07484)

Abstract:

>While transformers have begun to dominate many tasks in vision, applying them to large images is still computationally difficult. A large reason for this is that self-attention scales quadratically with the number of tokens, which in turn, scales quadratically with the image size. On larger images (e.g., 1080p), over 60% of the total computation in the network is spent solely on creating and applying attention matrices. We take a step toward solving this issue by introducing Hydra …

attention hydra machinelearning meta meta ai standard