LIMoE: Learning Multiple Modalities with One Sparse Mixture of Experts Model

Posted by Basil Mustafa, Research Software Engineer and Carlos Riquelme, Research Scientist, Google Research, Brain team

Sparse models stand out among the most promising approaches for the future of deep learning. Instead of every part of a model processing every input (“dense” modeling), sparse models employing conditional computation learn to route individual inputs to different “experts” in a potentially huge network. This has many benefits. First, model size can increase while keeping computational cost constant — an effective and environmentally …

computer vision deep learning experts image-classification learning machine learning mixture of experts multimodal learning nlp