[D] Is there MoE implemented for less than 1B total parameters?

I have a couple of questions:
1) Which is the currently widely used well performing MoE (Is it sparse moe?)

2) How does DeepSpeeds MoE fit in this picture? How well does it perform relative to others? Does any recent performant architectures adopt it?

3) Has anyone tried Sparse MoE for encoder-decoder model, say like for flan t5? Does it work similarly well? If so Why hasn't it been popular like deocder only variants?

4) Has anyone tried MoE for …

architectures machinelearning moe parameters questions total