Correlation-guided Query-Dependency Calibration in Video Representation Learning for Temporal Grounding

arXiv:2311.08835v3 Announce Type: replace
Abstract: Video Temporal Grounding is to identify specific moments or highlights from a video corresponding to textual descriptions. Typical approaches in temporal grounding treat all video clips equally during the encoding process regardless of their semantic relevance with the text query. Therefore, we propose Correlation-Guided DEtection TRansformer(CG-DETR), exploring to provide clues for query-associated video clips within the cross-modal attention. First, we design an adaptive cross-attention with dummy tokens. Dummy tokens conditioned by text query take portions …

abstract arxiv correlation cs.cv detection encoding highlights identify moments process query representation representation learning semantic temporal text textual type video