[Project] Sparse Mixture of Experts Language Model from Scratch in less than 300 lines of Python + pytorch

Hi all,I implemented a sparse mixture of experts language model (basically a tiny version of what's used in Mixtral, Grok-1 and supposedly GPT-4) from scratch in pure pytorch and trained it on tiny Shakespeare. This is based largely on makemore from Andrej Karpathy (an autoregressive character-level decoder only transformer model). My goal is for this to be a hackable implementation that people use to understand how this really works and improve upon. I foresee more and more of these models …

experts gpt gpt-4 grok language language model machinelearning mixtral mixture of experts project python pytorch scratch