[P] Advice regarding MoE and Mamba implementations

Hi everyone,

I'm diving into my Master's Thesis and need some guidance. The core of my work is to linearize a complex function that's riddled with memory effects. While the Transformer architecture has been explored in literature, I'm considering taking a fresh angle with either a Mamba architecture or spicing up the Transformer with a MoE (Mixture of Experts) approach. Moe-Mamba is also on the table.

The thing is: it's the first time I'm actually working with these architectures, so …

advice architecture core effects function guidance literature machinelearning mamba master memory moe thesis transformer transformer architecture work