BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences

arXiv:2403.09347v1 Announce Type: cross
Abstract: Effective attention modules have played a crucial role in the success of Transformer-based large language models (LLMs), but the quadratic time and memory complexities of these attention modules also pose a challenge when processing long sequences. One potential solution for the long sequence problem is to utilize distributed clusters to parallelize the computation of attention modules across multiple devices (e.g., GPUs). However, adopting a distributed approach inevitably introduces extra memory overheads to store local attention …

abstract arxiv attention challenge complexities cs.dc cs.lg distributed framework language language models large language large language models llms memory modules processing role solution success transformer type