Shedding the Bits: Pushing the Boundaries of Quantization with Minifloats on FPGAs

arXiv:2311.12359v2 Announce Type: replace-cross
Abstract: Post-training quantization (PTQ) is a powerful technique for model compression, reducing the numerical precision in neural networks without additional training overhead. Recent works have investigated adopting 8-bit floating-point formats(FP8) in the context of PTQ for model inference. However, floating-point formats smaller than 8 bits and their relative comparison in terms of accuracy-hardware cost with integers remains unexplored on FPGAs. In this work, we present minifloats, which are reduced-precision floating-point formats capable of further reducing the …

abstract arxiv compression context cs.ai cs.ar cs.cv cs.lg cs.pf fpgas however inference model inference networks neural networks numerical precision quantization replace training type