GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism

arXiv:2406.17145v1 Announce Type: cross
Abstract: Deep neural networks (DNNs) continue to grow rapidly in size, making them infeasible to train on a single device. Pipeline parallelism is commonly used in existing DNN systems to support large-scale DNN training by partitioning a DNN into multiple stages, which concurrently perform DNN training for different micro-batches in a pipeline fashion. However, existing pipeline-parallel approaches only consider sequential pipeline stages and thus ignore the topology of a DNN, resulting in missed model-parallel opportunities. This …

abstract arxiv cs.ai cs.dc cs.lg device dnn graph grow improving making multiple networks neural networks partitioning performance pipeline scalability scale stages support systems them train training type