[P] Don't have enough GPU to train Mixtral? Why not try LLaMA-MoE~

LLaMA-MoE is a series of open-sourced Mixture-of-Expert (MoE) models based on LLaMA and SlimPajama. We build LLaMA-MoE with the following two steps:

1. Partition LLaMA's FFNs into sparse experts and insert top-K gate for each layer of experts.
2. Continually pre-train the initialized MoE model with an optimized data sampling weights from Sheared LLaMA and filtered datasets from SlimPajama.

If you don't have plenty of computing resources to train Mixtral, you may want to try LLaMA-MoE for downstream researches.

Check …

build expert experts gate gpu layer llama machinelearning mixtral moe series train