Self attention with better complexity?

I work with transformers for many years and I know those models quite well. I use mostly LayoutLM.

Those lase weeks I looked at papers trying to reduce self attention complexity. The first was LongFormer. As I love the idea in the paper, I think the implementation is currently impossible as it would need sparse tensors. We tried those at work and have no speedup if the tensor is not VERY sparse. If you have a good way to deal …

attention complexity deal deeplearning good implementation layoutlm love paper reduce tensor think transformers work