Multi-Dimensional Model Compression of Vision Transformer. (arXiv:2201.00043v1 [cs.CV])

Vision transformers (ViT) have recently attracted considerable attentions,
but the huge computational cost remains an issue for practical deployment.
Previous ViT pruning methods tend to prune the model along one dimension
solely, which may suffer from excessive reduction and lead to sub-optimal model
quality. In contrast, we advocate a multi-dimensional ViT compression paradigm,
and propose to harness the redundancy reduction from attention head, neuron and
sequence dimensions jointly. We firstly propose a statistical dependence based
pruning criterion that is generalizable …

arxiv compression cv transformer