TAP: The Attention Patch for Cross-Modal Knowledge Transfer from Unlabeled Modality

arXiv:2302.02224v2 Announce Type: replace
Abstract: This paper addresses a cross-modal learning framework, where the objective is to enhance the performance of supervised learning in the primary modality using an unlabeled, unpaired secondary modality. Taking a probabilistic approach for missing information estimation, we show that the extra information contained in the secondary modality can be estimated via Nadaraya-Watson (NW) kernel regression, which can further be expressed as a kernelized cross-attention module (under linear transformation). Our results lay the foundations for introducing …

abstract arxiv attention cs.lg extra framework information knowledge modal paper performance show stat.ml supervised learning transfer type