Is Flash Attention Stable?

arXiv:2405.02803v1 Announce Type: new
Abstract: Training large-scale machine learning models poses distinct system challenges, given both the size and complexity of today's workloads. Recently, many organizations training state-of-the-art Generative AI models have reported cases of instability during training, often taking the form of loss spikes. Numeric deviation has emerged as a potential cause of this training instability, although quantifying this is especially challenging given the costly nature of training runs. In this work, we develop a principled approach to understanding …

abstract ai models art arxiv attention cases challenges complexity cs.dc cs.lg deviation flash form generative generative ai models loss machine machine learning machine learning models organizations scale state training type workloads