T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining

arXiv:2404.17806v1 Announce Type: cross
Abstract: Contrastive language-audio pretraining~(CLAP) has been developed to align the representations of audio and language, achieving remarkable performance in retrieval and classification tasks. However, current CLAP struggles to capture temporal information within audio and text features, presenting substantial limitations for tasks such as audio retrieval and generation. To address this gap, we introduce T-CLAP, a temporal-enhanced CLAP model. We use Large Language Models~(LLMs) and mixed-up strategies to generate temporal-contrastive captions for audio clips from extensive audio-text …

abstract arxiv audio clap classification cs.lg cs.sd current eess.as features gap however information language limitations performance presenting pretraining retrieval tasks temporal text type