Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression

arXiv:2403.15835v1 Announce Type: new
Abstract: Recent Vision Transformer Compression (VTC) works mainly follow a two-stage scheme, where the importance score of each model unit is first evaluated or preset in each submodule, followed by the sparsity score evaluation according to the target sparsity constraint. Such a separate evaluation process induces the gap between importance and sparsity score distributions, thus causing high search costs for VTC. In this work, for the first time, we investigate how to integrate the evaluations of …

abstract arxiv compression cs.cv evaluation importance search sparsity stage transformer type vision