Mixtral of Experts

Mixtral of Experts

The Mixtral paper is out, exactly a month after the release of the Mixtral 8x7B model itself. Thanks to the paper I now have a reasonable understanding of how a mixture of experts model works: each layer has 8 available blocks, but a router model selects two out of those eight for each token passing through that layer and combines their output. "As a result, each token has access to 47B parameters, but only uses 13B active …

ai experts generativeai layer llms mistral mixtral mixtral 8x7b mixture of experts paper release token understanding