Accelerating ViT Inference on FPGA through Static and Dynamic Pruning

arXiv:2403.14047v1 Announce Type: cross
Abstract: Vision Transformers (ViTs) have achieved state-of-the-art accuracy on various computer vision tasks. However, their high computational complexity prevents them from being applied to many real-world applications. Weight and token pruning are two well-known methods for reducing complexity: weight pruning reduces the model size and associated computational demands, while token pruning further dynamically reduces the computation based on the input. Combining these two techniques should significantly reduce computation complexity and model size; however, naively integrating them …

abstract accuracy applications art arxiv complexity computational computer computer vision cs.ar cs.cv cs.dc dynamic fpga however inference pruning state tasks them through token transformers type vision vision transformers vit world