EMC$^2$: Efficient MCMC Negative Sampling for Contrastive Learning with Global Convergence

arXiv:2404.10575v1 Announce Type: new
Abstract: A key challenge in contrastive learning is to generate negative samples from a large sample set to contrast with positive samples, for learning better encoding of the data. These negative samples often follow a softmax distribution which are dynamically updated during the training process. However, sampling from this distribution is non-trivial due to the high computational costs in computing the partition function. In this paper, we propose an Efficient Markov Chain Monte Carlo negative sampling …

abstract arxiv challenge contrast convergence cs.ai cs.cv cs.lg data distribution encoding generate global key math.oc mcmc negative positive process sample samples sampling set softmax training type