OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks. (arXiv:2210.12924v2 [cs.LG] UPDATED)

The size of deep neural networks has grown exponentially in recent years.
Unfortunately, hardware devices have not kept pace with the rapidly increasing
memory requirements. To cope with this, researchers have turned to techniques
such as spilling and recomputation, which increase training time, or reduced
precision and model pruning, which can affect model accuracy. We present OLLA,
an algorithm that optimizes the lifetime and memory location of the tensors
used to train neural networks. Our method reduces the memory usage …

arrays arxiv location memory networks neural networks reduce