Accelerating Transformer Pre-Training with 2:4 Sparsity

arXiv:2404.01847v1 Announce Type: new
Abstract: Training large Transformers is slow, but recent innovations on GPU architecture gives us an advantage. NVIDIA Ampere GPUs can execute a fine-grained 2:4 sparse matrix multiplication twice as fast as its dense equivalent. In the light of this property, we comprehensively investigate the feasibility of accelerating feed-forward networks (FFNs) of Transformers in pre-training. First, we define a "flip rate" to monitor the stability of a 2:4 training process. Utilizing this metric, we suggest two techniques …

abstract ampere architecture arxiv cs.lg fine-grained gpu gpus innovations light matrix matrix multiplication networks nvidia pre-training property sparsity training transformer transformers type