[N] MIT-IBM Watson AI Lab releases MoLM suite with three small sparse MoE models, the largest of which (8B params with 700M experts) performs on par with Pythia 2.8B while its throughput is comparable to Pythia 1.4B

Paper: [https://arxiv.org/abs/2306.04640](https://arxiv.org/abs/2306.04640)

GitHub: [https://github.com/ibm/moduleformer](https://github.com/ibm/moduleformer) (under Apache 2.0)

Twitter thread: [https://twitter.com/Yikang\_Shen/status/1702041129267388678](https://twitter.com/Yikang_Shen/status/1702041129267388678)

Abstract:

>Large Language Models (LLMs) have achieved remarkable results. However, existing models are expensive to train and deploy, and it is also difficult to expand their knowledge beyond pre-training data without forgetting previous knowledge. This paper proposes a new neural network architecture, ModuleFormer, that leverages modularity to improve the efficiency and flexibility of large language models. ModuleFormer is based on the Sparse Mixture of Experts (SMoE). Unlike the previous SMoE-based …

abstract architecture beyond data deploy efficiency flexibility knowledge language language models large language large language models llms machinelearning network network architecture neural network paper pre-training training training data