DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers

arXiv:2403.10266v1 Announce Type: cross
Abstract: Scaling large models with long sequences across applications like language generation, video generation and multimodal tasks requires efficient sequence parallelism. However, existing sequence parallelism methods all assume a single sequence dimension and fail to adapt to multi-dimensional transformer architectures that perform attention calculations across different dimensions. This paper introduces Dynamic Sequence Parallelism (DSP), a novel approach to enable efficient sequence parallelism for multi-dimensional transformer models. The key idea is to dynamically switch the parallelism dimension …

abstract adapt applications architectures arxiv attention cs.dc cs.lg dimensions dsp dynamic however language language generation large models multimodal paper scaling tasks transformer transformers type video video generation