IM-Unpack: Training and Inference with Arbitrarily Low Precision Integers

arXiv:2403.07339v1 Announce Type: new
Abstract: GEneral Matrix Multiply (GEMM) is a central operation in deep learning and corresponds to the largest chunk of the compute footprint. Therefore, improving its efficiency is an active topic of ongoing research. A popular strategy is the use of low bit-width integers to approximate the original entries in a matrix. This allows efficiency gains, but often requires sophisticated techniques to control the rounding error incurred. In this work, we first verify/check that when the low …

abstract arxiv compute cs.cl cs.cv cs.lg deep learning efficiency general inference integers low matrix popular precision research strategy training type