TransFusion: Contrastive Learning with Transformers

arXiv:2403.18681v1 Announce Type: new
Abstract: This paper proposes a novel framework, TransFusion, designed to make the process of contrastive learning more analytical and explainable. TransFusion consists of attention blocks whose softmax being replaced by ReLU, and its final block's weighted-sum operation is truncated to leave the adjacency matrix as the output. The model is trained by minimizing the Jensen-Shannon Divergence between its output and the target affinity matrix, which indicates whether each pair of samples belongs to the same or …

abstract arxiv attention block cs.ai cs.lg framework matrix novel paper process relu softmax transformers type