Transformer based Multitask Learning for Image Captioning and Object Detection

arXiv:2403.06292v1 Announce Type: new
Abstract: In several real-world scenarios like autonomous navigation and mobility, to obtain a better visual understanding of the surroundings, image captioning and object detection play a crucial role. This work introduces a novel multitask learning framework that combines image captioning and object detection into a joint model. We propose TICOD, Transformer-based Image Captioning and Object detection model for jointly training both tasks by combining the losses obtained from image captioning and object detection networks. By leveraging …

abstract arxiv autonomous captioning cs.cl cs.cv detection framework image mobility multitask learning navigation novel object role transformer type understanding visual work world