[P] Multihead Mixture of Experts - Implementation of dense subtoken routing suggested in https://arxiv.org/pdf/2404.15045

My friend implemented the method of Multihead Mixture of Experts in this arxiv paper [https://arxiv.org/pdf/2404.15045](https://arxiv.org/pdf/2404.15045) and he wanted me to share it with you!

https://github.com/lhallee/Multi_Head_Mixture_of_Experts__MH-MOE

Try it out. Let me know what you think and I will pass it on to him.

machinelearning think will