DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Models

arXiv:2404.08079v1 Announce Type: new
Abstract: Recent advances in decentralized deep learning algorithms have demonstrated cutting-edge performance on various tasks with large pre-trained models. However, a pivotal prerequisite for achieving this level of competitiveness is the significant communication and computation overheads when updating these models, which prohibits the applications of them to real-world scenarios. To address this issue, drawing inspiration from advanced model merging techniques without requiring additional training, we introduce the Decentralized Iterative Merging-And-Training (DIMAT) paradigm--a novel decentralized deep learning …

abstract advances algorithms applications arxiv communication computation cs.cv cs.lg decentralized deep learning deep learning algorithms edge however iterative math.oc merging performance pivotal pre-trained models tasks them training type world