[R] LoRA-MoE: Training and inferencing MoE models like Mixtral 8x7B like a 7B param model

MoE models like Mixtral 8x7B use 8 distinct experts of dense matrices in fully connected blocks, two of which are selected by a router network and their outputs are combined, while processing a token.

Since only two of the groups need to be loaded to the memory, while others remain offloaded, this requires the model to use 12.9B parameters out of the 46.7B total parameters at any point.

I'm wondering to bring the parameters down to almost the same level …

experts inferencing lora machinelearning mixtral mixtral 8x7b moe network processing token training