Detecting Concrete Visual Tokens for Multimodal Machine Translation

arXiv:2403.03075v1 Announce Type: new
Abstract: The challenge of visual grounding and masking in multimodal machine translation (MMT) systems has encouraged varying approaches to the detection and selection of visually-grounded text tokens for masking. We introduce new methods for detection of visually and contextually relevant (concrete) tokens from source sentences, including detection with natural language processing (NLP), detection with object detection, and a joint detection-verification technique. We also introduce new methods for selection of detected tokens, including shortest $n$ tokens, longest …

abstract arxiv challenge concrete cs.cl detection machine machine translation masking multimodal systems text tokens translation type visual