Sine Activated Low-Rank Matrices for Parameter Efficient Learning

arXiv:2403.19243v1 Announce Type: new
Abstract: Low-rank decomposition has emerged as a vital tool for enhancing parameter efficiency in neural network architectures, gaining traction across diverse applications in machine learning. These techniques significantly lower the number of parameters, striking a balance between compactness and performance. However, a common challenge has been the compromise between parameter efficiency and the accuracy of the model, where reduced parameters often lead to diminished accuracy compared to their full-rank counterparts. In this work, we propose a …

abstract applications architectures arxiv balance challenge cs.cv cs.lg cs.ne diverse diverse applications efficiency however low machine machine learning network neural network parameters performance tool type vital