Listen Then See: Video Alignment with Speaker Attention

arXiv:2404.13530v1 Announce Type: cross
Abstract: Video-based Question Answering (Video QA) is a challenging task and becomes even more intricate when addressing Socially Intelligent Question Answering (SIQA). SIQA requires context understanding, temporal reasoning, and the integration of multimodal information, but in addition, it requires processing nuanced human behavior. Furthermore, the complexities involved are exacerbated by the dominance of the primary modality (text) over the others. Thus, there is a need to help the task's secondary modalities to work in tandem with …

alignment arxiv attention cs.cl cs.cv cs.lg speaker type video