all AI news
[R] ZeRO++: Extremely Efficient Collective Communication for Giant Model Training - Microsoft 2023
July 6, 2023, 3:57 p.m. | /u/Singularian2501
Machine Learning www.reddit.com
Github: [https://github.com/microsoft/DeepSpeed](https://github.com/microsoft/DeepSpeed)
Abstract:
>Zero Redundancy Optimizer (ZeRO) has been used to train a wide range of large language models on massive GPUs clusters due to its ease of use, efficiency, and good scalability. However, when training on low-bandwidth clusters, or at scale which forces batch size per GPU to be small, ZeRO's effective throughput is limited because of high communication volume from gathering weights in forward pass, backward pass, and averaging gradients. This paper introduces three communication volume …
abstract communication efficiency good gpu gpus language language models large language large language models low machinelearning massive per redundancy scalability scale small training
More from www.reddit.com / Machine Learning
Jobs in AI, ML, Big Data
Founding AI Engineer, Agents
@ Occam AI | New York
AI Engineer Intern, Agents
@ Occam AI | US
AI Research Scientist
@ Vara | Berlin, Germany and Remote
Data Architect
@ University of Texas at Austin | Austin, TX
Data ETL Engineer
@ University of Texas at Austin | Austin, TX
Lead GNSS Data Scientist
@ Lurra Systems | Melbourne