Noise contrastive estimation with soft targets for conditional models

arXiv:2404.14076v1 Announce Type: new
Abstract: Soft targets combined with the cross-entropy loss have shown to improve generalization performance of deep neural networks on supervised classification tasks. The standard cross-entropy loss however assumes data to be categorically distributed, which may often not be the case in practice. In contrast, InfoNCE does not rely on such an explicit assumption but instead implicitly estimates the true conditional through negative sampling. Unfortunately, it cannot be combined with soft targets in its standard formulation, hindering …

abstract arxiv case classification contrast cross-entropy cs.cv cs.lg data distributed entropy however loss networks neural networks noise performance practice standard stat.ml targets tasks type