‘MrsFormer’ Employs a Nove Multiresolution-Head Attention Mechanism to Cut Transformers’ Compute and Memory Costs

In the new paper Transformers with Multiresolution Attention Heads (currently under double-blind review for ICLR 2023), researchers propose MrsFormer, a novel transformer architecture that uses Multiresolution-head Attention to approximate output sequences and significantly reduces head redundancy without sacrificing accuracy.

ai artificial intelligence attention attention mechanisms compute costs deep-neural-networks head machine learning machine learning & data science memory ml research technology transformers