Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency

arXiv:2303.11525v3 Announce Type: replace
Abstract: Recent research has focused on weight sparsity in neural network training to reduce FLOPs, aiming for improved efficiency (test accuracy w.r.t training FLOPs). However, sparse weight training often sacrifices accuracy, requiring extended training schedules to attain the accuracy of dense models. In contrast, our approach, Sparse Iso-FLOP Transformations (Sparse-IFT), uses sparsity to improve accuracy while maintaining dense model FLOPs. Using a single hyperparameter (i.e., sparsity level), Sparse-IFTs efficiently replace dense layers, expanding the search space …

arxiv cs.cl cs.cv cs.lg efficiency iso training type