CromSS: Cross-modal pre-training with noisy labels for remote sensing image segmentation

arXiv:2405.01217v1 Announce Type: new
Abstract: We study the potential of noisy labels y to pretrain semantic segmentation models in a multi-modal learning framework for geospatial applications. Specifically, we propose a novel Cross-modal Sample Selection method (CromSS) that utilizes the class distributions P^{(d)}(x,c) over pixels x and classes c modelled by multiple sensors/modalities d of a given geospatial scene. Consistency of predictions across sensors $d$ is jointly informed by the entropy of P^{(d)}(x,c). Noisy label sampling we determine by the confidence …

abstract applications arxiv class cs.cv framework geospatial image labels modal multi-modal novel pixels pre-training sample segmentation semantic sensing study training type