Bidirectional Generative Pre-training for Improving Time Series Representation Learning

arXiv:2402.09558v1 Announce Type: cross
Abstract: Learning time-series representations for discriminative tasks has been a long-standing challenge. Current pre-training methods are limited in either unidirectional next-token prediction or randomly masked token prediction. We propose a novel architecture called Bidirectional Timely Generative Pre-trained Transformer (BiTimelyGPT), which pre-trains on time-series data by both next-token and previous-token predictions in alternating transformer layers. This pre-training task preserves original distribution and data shapes of the time-series. Additionally, the full-rank forward and backward attention matrices exhibit more …

abstract architecture arxiv challenge cs.ai cs.lg current data generative generative pre-trained transformer next novel prediction pre-training representation representation learning series tasks time series token training trains transformer type