Schr\"odinger's FP: Dynamic Adaptation of Floating-Point Containers for Deep Learning Training. (arXiv:2204.13666v1 [cs.LG])

We introduce a software-hardware co-design approach to reduce memory traffic
and footprint during training with BFloat16 or FP32 boosting energy efficiency
and execution time performance. We introduce methods to dynamically adjust the
size and format of the floating-point containers used to store activations and
weights during training. The different value distributions lead us to different
approaches for exponents and mantissas. Gecko exploits the favourable exponent
distribution with a loss-less delta encoding approach to reduce the total
exponent footprint by up …

arxiv containers deep learning deep learning training learning training