Ultimate Guide To Scaling ML Models - Megatron-LM | ZeRO | DeepSpeed | Mixed Precision

🚀 Sign up for AssemblyAI's speech API using my link 🚀
https://www.assemblyai.com/?utm_source=youtube&utm_medium=social&utm_campaign=theaiepiphany

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
https://discord.gg/peBrCpheKE

In this video I show you what it takes to scale ML models up to trillions of parameters!

I cover the fundamental ideas behind all of the recent big ML models you must have heard of like Meta's OPT-175B, BigScience BLOOM 176B, EleutherAI's GPT-NeoX-20B, GPT-J, OpenAI's GPT-3, Google's PaLM, DeepMind's Chinchilla/Gopher models, etc.

I cover the ideas of data parallelism, model/pipeline …

deepspeed guide mixed ml models precision scaling