[R] QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models - Institute of Science and Technology Austria (ISTA) 2023 - Can compress the 1.6 trillion parameter SwitchTransformer-c2048 model to less than 160GB (20x compression, 0.8 bits per

Paper: [https://arxiv.org/abs/2310.16795](https://arxiv.org/abs/2310.16795)

Github: [https://github.com/ist-daslab/qmoe](https://github.com/ist-daslab/qmoe)

Abstract:

>Mixture-of-Experts (MoE) architectures offer a general solution to the high inference costs of large language models (LLMs) via sparse routing, bringing faster and more accurate models, at the cost of massive parameter counts. For example, the SwitchTransformer-c2048 model has 1.6 trillion parameters, requiring 3.2TB of accelerator memory to run efficiently, which makes practical deployment challenging and expensive. In this paper, we present a solution to this memory problem, in form of a new compression and …

abstract accelerator architectures cost costs deployment example experts faster general inference inference costs language language models large language large language models llms machinelearning massive memory moe parameters practical routing solution