Conformer: Embedding Continuous Attention in Vision Transformer for Weather Forecasting

arXiv:2402.17966v1 Announce Type: new
Abstract: Operational weather forecasting system relies on computationally expensive physics-based models. Although Transformers-based models have shown remarkable potential in weather forecasting, Transformers are discrete models which limit their ability to learn the continuous spatio-temporal features of the dynamical weather system. We address this issue with Conformer, a spatio-temporal Continuous Vision Transformer for weather forecasting. Conformer is designed to learn the continuous weather evolution over time by implementing continuity in the multi-head attention mechanism. The attention mechanism …

abstract arxiv attention continuous cs.lg embedding features forecasting issue learn physics temporal transformer transformers type vision weather weather forecasting