LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking

arXiv:2403.04303v1 Announce Type: new
Abstract: Deep learning models, particularly those based on transformers, often employ numerous stacked structures, which possess identical architectures and perform similar functions. While effective, this stacking paradigm leads to a substantial increase in the number of parameters, posing challenges for practical applications. In today's landscape of increasingly large models, stacking depth can even reach dozens, further exacerbating this issue. To mitigate this problem, we introduce LORS (LOw-rank Residual Structure). LORS allows stacked modules to share the …

abstract applications architectures arxiv challenges cs.cv deep learning functions landscape leads low network paradigm parameters practical residual transformers type