Distributed Parallel Training: Data Parallelism and Model Parallelism

How to scale out training large models like GPT-3 & DALL-E 2 in PyTorch

Recent years have witnessed exponential growth in the scale of distributed parallel training and the size of deep learning models. In particular, Transformer-based language models have been stealing the show. The notorious GPT-3 blew out with 175 billion parameters and 96 attention layers with a 3.2 M batch size and 499 billion words. Exactly half a year later, Google published …

data distributed distributed-training model-parallelism pytorch training