The Hidden Power of Pure 16-bit Floating-Point Neural Networks

arXiv:2301.12809v2 Announce Type: replace
Abstract: Lowering the precision of neural networks from the prevalent 32-bit precision has long been considered harmful to performance, despite the gain in space and time. Many works propose various techniques to implement half-precision neural networks, but none study pure 16-bit settings. This paper investigates the unexpected performance gain of pure 16-bit neural networks over the 32-bit networks in classification tasks. We present extensive experimental results that favorably compare various 16-bit neural networks' performance to those …

16-bit abstract arxiv cs.ai cs.lg cs.pf hidden networks neural networks paper performance power precision space space and time study type