MuLTI: Efficient Video-and-Language Understanding with Text-Guided MultiWay-Sampler and Multiple Choice Modeling

arXiv:2303.05707v2 Announce Type: replace
Abstract: Video-and-language understanding has a variety of applications in the industry, such as video question answering, text-video retrieval, and multi-label classification. Existing video-and-language understanding methods generally adopt heavy multi-modal encoders and feature fusion modules, which consume high computational costs. Specially, they have difficulty dealing with dense video frames or long text prevalent in industrial applications. This paper proposes MuLTI, a highly accurate and efficient video-and-language understanding model that achieves efficient and effective feature fusion and rapid …

abstract applications arxiv classification computational costs cs.cl cs.cv cs.mm feature fusion industry language language understanding modal modeling modules multi-modal multiple question question answering retrieval text type understanding video