Self-Supervised Contrastive Pre-Training for Multivariate Point Processes

Self-supervision is one of the hallmarks of representation learning in the increasingly popular suite of foundation models including large language models such as BERT and GPT-3, but it has not been pursued in the context of multivariate event streams, to the best of our knowledge. We introduce a new paradigm for self-supervised learning for multivariate point processes using a transformer encoder. Specifically, we design a novel pre-training strategy for the encoder where we not only mask random event epochs but …

bert best of context cs.lg event foundation gpt gpt-3 knowledge language language models large language large language models multivariate new paradigm paradigm popular pre-training processes representation representation learning self-supervised learning supervised learning supervision training