LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memory

arXiv:2404.11163v2 Announce Type: replace
Abstract: Transformer models have been successful in various sequence processing tasks, but the self-attention mechanism's computational cost limits its practicality for long sequences. Although there are existing attention variants that improve computational efficiency, they have a limited ability to abstract global information effectively based on their hand-crafted mixing strategies. On the other hand, state-space models (SSMs) are tailored for long sequences but cannot capture complicated local information. Therefore, the combination of them as a unified token …

abstract arxiv attention computational cost cs.lg efficiency global information memory modeling processing quantization self-attention tasks transformer transformer models type variants vector