TerViT: An Efficient Ternary Vision Transformer. (arXiv:2201.08050v1 [cs.CV])

Vision transformers (ViTs) have demonstrated great potential in various
visual tasks, but suffer from expensive computational and memory cost problems
when deployed on resource-constrained devices. In this paper, we introduce a
ternary vision transformer (TerViT) to ternarize the weights in ViTs, which are
challenged by the large loss surface gap between real-valued and ternary
parameters. To address the issue, we introduce a progressive training scheme by
first training 8-bit transformers and then TerViT, and achieve a better
optimization than conventional …

arxiv cv transformer vision