Liam Fedus & Barret Zoph - AI scaling with mixture of expert models

Liam Fedus and Barrett Zoph, research scientists at Google Brain, came on the podcast to talk about AI scaling, sparsity and the present and future of MoE models.

Intro music:
➞ Artist: Ron Gelinas
➞ Track Title: Daybreak Chill Blend (original mix)
➞ Link to Track: https://youtu.be/d8Y2sKIgFWc

0:00 Intro
2:15 Guests’ backgrounds
8:00 Understanding specialization
13:45 Speculations for the future
21:45 Switch transformer versus dense net
27:30 More interpretable models
33:30 Assumptions and biology
39:15 Wrap-up

ai expert scaling