LSTP: Language-guided Spatial-Temporal Prompt Learning for Long-form Video-Text Understanding

arXiv:2402.16050v1 Announce Type: new
Abstract: Despite progress in video-language modeling, the computational challenge of interpreting long-form videos in response to task-specific linguistic queries persists, largely due to the complexity of high-dimensional video data and the misalignment between language and visual cues over space and time. To tackle this issue, we introduce a novel approach called Language-guided Spatial-Temporal Prompt Learning (LSTP). This approach features two key components: a Temporal Prompt Sampler (TPS) with optical flow prior that leverages temporal information to …

abstract arxiv challenge complexity computational cs.cl cs.cv data form issue language modeling progress prompt prompt learning queries space space and time spatial temporal text text understanding type understanding video video data videos visual visual cues