THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression

arXiv:2302.08545v2 Announce Type: replace
Abstract: Deep neural networks (DNNs) are the de facto standard for essential use cases, such as image classification, computer vision, and natural language processing. As DNNs and datasets get larger, they require distributed training on increasingly larger clusters. A main bottleneck is the resulting communication overhead where workers exchange model updates (i.e., gradients) on a per-round basis. To address this bottleneck and accelerate training, a widely-deployed approach is compression. However, previous deployments often apply bi-directional compression …

abstract and natural language processing arxiv cases classification communication compression computer computer vision cs.ai cs.lg cs.ni datasets deep learning distributed image language language processing natural natural language natural language processing networks neural networks processing standard tensor training type use cases vision