Centered Masking for Language-Image Pre-Training

arXiv:2403.15837v1 Announce Type: cross
Abstract: We introduce Gaussian masking for Language-Image Pre-Training (GLIP) a novel, straightforward, and effective technique for masking image patches during pre-training of a vision-language model. GLIP builds on Fast Language-Image Pre-Training (FLIP), which randomly masks image patches while training a CLIP model. GLIP replaces random masking with centered masking, that uses a Gaussian distribution and is inspired by the importance of image patches at the center of the image. GLIP retains the same computational savings as …

abstract arxiv clip cs.cl cs.cv cs.lg image language language model masking masks novel pre-training random training type vision