Google AI Introduces A Method For Automating Inter- And Intra-Operator Parallelism For Distributed Deep Learning

Web: https://www.reddit.com/r/artificial/comments/ull27x/google_ai_introduces_a_method_for_automating/

The memory capacity of single accelerators has swiftly overtaken the rapidly rising size of deep learning models in recent years. Earlier models, such as BERT (with a parameter size of 1GB), may scale across accelerators quickly by utilizing [data parallelism](https://en.wikipedia.org/wiki/Data_parallelism), which duplicates model weights across accelerators while splitting and distributing training data. Recent huge models, such as GPT-3 (with a parameter size of 175GB), can only be scaled through model parallelism training, which involves partitioning a single model across many …

ai artificial deep deep learning distributed google learning