Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training

arXiv:2404.08327v1 Announce Type: new
Abstract: In this paper, we introduce Saliency-Based Adaptive Masking (SBAM), a novel and cost-effective approach that significantly enhances the pre-training performance of Masked Image Modeling (MIM) approaches by prioritizing token salience. Our method provides robustness against variations in masking ratios, effectively mitigating the performance instability issues common in existing methods. This relaxes the sensitivity of MIM-based pre-training to masking ratios, which in turn allows us to propose an adaptive strategy for `tailored' masking ratios for each …

abstract arxiv cost cs.cv dynamics image masking modeling novel paper performance pre-training robustness token training type