Training great LLMs entirely from ground zero in the wilderness as a startup

Training great LLMs entirely from ground zero in the wilderness as a startup

Yi Tay has a really interesting perspective on training LLMs, having worked at Google Brain before co-founding an independent startup, Reka.

At Google the clusters are provided for you. On the outside, Yi finds himself bargaining for cluster resources from a wide range of vendors - and running into enormous variance in quality.

"We’ve seen clusters that range from passable (just annoying problems that are solvable with …

ai brain google google brain independent llms perspective reka startup training training llms