MoE: Revolutionizing Memory-Efficient Execution of Massive-Scale MoE Models

A research team from Institute of Science and Technology Austria (ISTA) and Neural Magic Inc. introduces the QMoE framework. This innovative framework offers an effective solution for accurately compressing massive MoEs and conducting swift compressed inference, reducing model sizes by 10–20×, achieving less than 1 bit per parameter.

ai artificial intelligence austria deep-neural-networks framework inference institute large language model machine learning machine learning & data science magic massive memory mixture of experts ml moe neural magic per research research team scale science science and technology solution swift team technology