Fuse after Align: Improving Face-Voice Association Learning via Multimodal Encoder

arXiv:2404.09509v1 Announce Type: new
Abstract: Today, there have been many achievements in learning the association between voice and face. However, most previous work models rely on cosine similarity or L2 distance to evaluate the likeness of voices and faces following contrastive learning, subsequently applied to retrieval and matching tasks. This method only considers the embeddings as high-dimensional vectors, utilizing a minimal scope of available information. This paper introduces a novel framework within an unsupervised setting for learning voice-face associations. By …

abstract arxiv association cosine cs.cv encoder face however improving likeness multimodal retrieval tasks type via voice voices work