480B LLM as 128x4B MoE? WHY?

Snowflake offers unique tech insights in new architecture designs of LLM, particular its new 128x3.66B Mixture of Expert (MoE) system. Dedicated, highly specialized LLMs for particular tasks.

Short introduction to MoE and then a comparison between different model architectures, followed up by a causal reasoning test (following test suite published by Stanford Univ).

Can a relatively small LLM, with below eg 5 Billion free trainable parameters, solve complex reasoning tasks. We evaluated this in my last video on PHI-3 MINI. …

architecture architectures causal comparison designs expert insights introduction llm llms moe reasoning snowflake stanford tasks tech tech insights test unique