Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs

arXiv:2405.03103v1 Announce Type: new
Abstract: Large language models (LLMs) have recently achieved state-of-the-art performance across various tasks, yet due to their large computational requirements, they struggle with strict latency and power demands. Deep neural network (DNN) quantization has traditionally addressed these limitations by converting models to low-precision integer formats. Yet recently alternative formats, such as Normal Float (NF4), have been shown to consistently increase model accuracy, albeit at the cost of increased chip area. In this work, we first conduct …

abstract art arxiv computational cs.cv cs.lg deep neural network dnn explore language language models large language large language models latency limitations llms low network neural network performance power precision quantization requirements state struggle students tasks type