LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models

arXiv:2206.09557v4 Announce Type: replace-cross
Abstract: Recent advances in self-supervised learning and the Transformer architecture have significantly improved natural language processing (NLP), achieving remarkably low perplexity. However, the growing size of NLP models introduces a memory wall problem during the generation phase. To mitigate this issue, recent efforts have focused on quantizing model weights to sub-4-bit precision while preserving full precision for activations, resulting in practical speed-ups during inference on a single GPU. However, these improvements primarily stem from reduced memory …

abstract advances architecture arxiv cs.cl cs.dc generative however inference issue language language models language processing low lut matrix matrix multiplication memory natural natural language natural language processing nlp nlp models perplexity processing scale self-supervised learning supervised learning transformer transformer architecture type