Early Transformers: A study on Efficient Training of Transformer Models through Early-Bird Lottery Tickets

arXiv:2405.02353v1 Announce Type: cross
Abstract: The training of Transformer models has revolutionized natural language processing and computer vision, but it remains a resource-intensive and time-consuming process. This paper investigates the applicability of the early-bird ticket hypothesis to optimize the training efficiency of Transformer models. We propose a methodology that combines iterative pruning, masked distance calculation, and selective retraining to identify early-bird tickets in various Transformer architectures, including ViT, Swin-T, GPT-2, and RoBERTa. Our experimental results demonstrate that early-bird tickets can …

abstract arxiv bird computer computer vision cs.cl cs.lg efficiency hypothesis language language processing lottery natural natural language natural language processing paper process processing study through tickets training transformer transformer models transformers type vision