Emerging Property of Masked Token for Effective Pre-training

arXiv:2404.08330v1 Announce Type: new
Abstract: Driven by the success of Masked Language Modeling (MLM), the realm of self-supervised learning for computer vision has been invigorated by the central role of Masked Image Modeling (MIM) in driving recent breakthroughs. Notwithstanding the achievements of MIM across various downstream tasks, its overall efficiency is occasionally hampered by the lengthy duration of the pre-training phase. This paper presents a perspective that the optimization of masked tokens as a means of addressing the prevailing issue. …

abstract arxiv computer computer vision cs.cv driving efficiency image language modeling pre-training property realm role self-supervised learning success supervised learning tasks token training type vision