[D] Question about Mixture of Experts in Transformers - Has anyone tried adding the router before the Multi Head Attention Blocks?

This question came up in our Friday paper club as we read the Mixtral 8x7B paper, and don't feel like we got a satisfying answer.

It seems like the argument for MOE is that you can let certain parts of the network specialize in certain domains or tasks. This also strikes me as a similar argument people make for having multi-head attention within the transformer block. Why would you only put the router in front of the Feed Forward Layers …

attention experts head machinelearning mixtral mixtral 8x7b mixture of experts moe paper question transformers