FP8 Quantization: The Power of the Exponent

arXiv:2208.09225v2 Announce Type: replace
Abstract: When quantizing neural networks for efficient inference, low-bit integers are the go-to format for efficiency. However, low-bit floating point numbers have an extra degree of freedom, assigning some bits to work on an exponential scale instead. This paper in-depth investigates this benefit of the floating point format for neural network inference. We detail the choices that can be made for the FP8 format, including the important choice of the number of bits for the mantissa …

abstract arxiv benefit cs.lg efficiency extra floating point format freedom inference integers low networks neural networks numbers paper power quantization scale type work