DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models

arXiv:2403.00818v1 Announce Type: cross
Abstract: Large language models (LLMs) face a daunting challenge due to the excessive computational and memory requirements of the commonly used Transformer architecture. While state space model (SSM) is a new type of foundational network architecture offering lower computational complexity, their performance has yet to fully rival that of Transformers. This paper introduces DenseSSM, a novel approach to enhance the flow of hidden information between layers in SSMs. By selectively integrating shallowlayer hidden states into deeper …

abstract architecture arxiv challenge complexity computational cs.cl cs.lg face hidden language language models large language large language models llms memory network network architecture performance requirements space state transformer transformer architecture type