TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages

arXiv:2402.16021v1 Announce Type: cross
Abstract: The capability to jointly process multi-modal information is becoming an essential task. However, the limited number of paired multi-modal data and the large computational requirements in multi-modal learning hinder the development. We propose a novel Tri-Modal Translation (TMT) model that translates between arbitrary modalities spanning speech, image, and text. We introduce a novel viewpoint, where we interpret different modalities as different languages, and treat multi-modal translation as a well-established machine translation problem. To this end, …

abstract arxiv capability computational cs.ai cs.cl cs.cv data development eess.as hinder image information languages modal multi-modal novel process processing requirements speech text translation type