Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matrices

arXiv:2403.14958v1 Announce Type: new
Abstract: As deep learning models exponentially increase in size, optimizers such as Adam encounter significant memory consumption challenges due to the storage of first and second moment data. Current memory-efficient methods like Adafactor and CAME often compromise accuracy with their matrix factorization techniques. Addressing this, we introduce Adapprox, a novel approach that employs randomized low-rank matrix approximation for a more effective and accurate approximation of Adam's second moment. Adapprox features an adaptive rank selection mechanism, finely …

abstract accuracy adam approximation arxiv challenges consumption cs.cl cs.lg current data deep learning factorization low math.oc matrix memory memory consumption optimization storage type via