A General and Efficient Training for Transformer via Token Expansion

arXiv:2404.00672v1 Announce Type: new
Abstract: The remarkable performance of Vision Transformers (ViTs) typically requires an extremely large training cost. Existing methods have attempted to accelerate the training of ViTs, yet typically disregard method universality with accuracy dropping. Meanwhile, they break the training consistency of the original transformers, including the consistency of hyper-parameters, architecture, and strategy, which prevents them from being widely applied to different Transformer networks. In this paper, we propose a novel token growth scheme Token Expansion (termed ToE) …

arxiv cs.ai cs.cl cs.cv cs.lg expansion general token training transformer type via