Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Servers. (arXiv:2202.01306v2 [cs.DC] UPDATED)

Deep neural networks (DNNs) have grown exponentially in size over the past
decade, leaving only those who have massive datacenter-based resources with the
ability to develop and train such models. One of the main challenges for the
long tail of researchers who might have only limited resources (e.g., a single
multi-GPU server) is limited GPU memory capacity compared to model size. The
problem is so acute that the memory requirement of training massive DNN models
can often exceed the aggregate …

arxiv capacity dnn gpu massive memory servers