Quantization of Large Language Models with an Overdetermined Basis

arXiv:2404.09737v1 Announce Type: new
Abstract: In this paper, we introduce an algorithm for data quantization based on the principles of Kashin representation. This approach hinges on decomposing any given vector, matrix, or tensor into two factors. The first factor maintains a small infinity norm, while the second exhibits a similarly constrained norm when multiplied by an orthogonal matrix. Surprisingly, the entries of factors after decomposition are well-concentrated around several peaks, which allows us to efficiently replace them with corresponding centroids …

abstract algorithm arxiv cs.cl cs.lg data language language models large language large language models matrix norm paper quantization representation small tensor type vector