OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition

arXiv:2312.00096v2 Announce Type: replace
Abstract: Due to the resource-intensive nature of training vision-language models on expansive video data, a majority of studies have centered on adapting pre-trained image-language models to the video domain. Dominant pipelines propose to tackle the visual discrepancies with additional temporal learners while overlooking the substantial discrepancy for web-scaled descriptive narratives and concise action category names, leading to less distinct semantic space and potential performance limitations. In this work, we prioritize the refinement of text knowledge to …

abstract arxiv cs.cv data domain general image knowledge language language models nature pipelines recognition studies temporal text training type video video data vision vision-language models visual