Characterizing the Accuracy - Efficiency Trade-off of Low-rank Decomposition in Language Models

arXiv:2405.06626v1 Announce Type: new
Abstract: Large language models (LLMs) have emerged and presented their general problem-solving capabilities with one model. However, the model size has increased dramatically with billions of parameters to enable such broad problem-solving capabilities. In addition, due to the dominance of matrix-matrix and matrix-vector multiplications in LLMs, the compute-to-model size ratio is significantly lower than that of CNNs. This shift pushes LLMs from a computation-bound regime to a memory-bound regime. Therefore, optimizing the memory footprint and traffic …

abstract accuracy arxiv capabilities cs.cl cs.lg efficiency general however language language models large language large language models llms low matrix one model parameters problem-solving trade trade-off type vector