Communication Efficient Distributed Training with Distributed Lion

arXiv:2404.00438v1 Announce Type: cross
Abstract: The Lion optimizer has been a promising competitor with the AdamW for training large AI models, with advantages on memory, computation, and sample efficiency. In this paper, we introduce Distributed Lion, an innovative adaptation of Lion for distributed training environments. Leveraging the sign operator in Lion, our Distributed Lion only requires communicating binary or lower-precision vectors between workers to the center server, significantly reducing the communication cost. Our theoretical analysis confirms Distributed Lion's convergence properties. …

abstract advantages ai models arxiv communication computation cs.ai cs.dc cs.lg distributed efficiency environments math.oc memory paper sample stat.ml training type