Reducing Activation Recomputation in Large Transformer Models. (arXiv:2205.05198v1 [cs.LG])

Web: http://arxiv.org/abs/2205.05198

Training large transformer models is one of the most important computational
challenges of modern AI. In this paper, we show how to significantly accelerate
training of large transformer models by reducing activation recomputation.
Activation recomputation is commonly used to work around memory capacity
constraints. Rather than storing activations for backpropagation, they are
traditionally recomputed, which saves memory but adds redundant compute. In
this work, we show most of this redundant compute is unnecessary because we can
reduce memory consumption sufficiently …

arxiv models transformer transformer models