[D] how to accelerate ViT models more faster

I have conducted experiments and examples on accelerating ViT (Vision Transformer) using methods such as TensorRT, FasterTransformer, and xFormers. The experiments were conducted using a single A100 as a baseline. - [https://github.com/bnabis93/vision-language-examples/tree/main/acceleration](https://github.com/bnabis93/vision-language-examples/tree/main/acceleration)


In xFormers, I tried applying sparse attention and memory-efficient attention to ViT, but there was an issue where the speed actually decreased. Therefore, I excluded those results.


Generally, just performing TensorRT conversion significantly improves latency. In the case of faster transformer, optimized kernels are not provided in fp32, …

attention case conversion faster issue latency machinelearning memory speed tensorrt transformer vit