Distinctive Image Captioning: Leveraging Ground Truth Captions in CLIP Guided Reinforcement Learning

arXiv:2402.13936v1 Announce Type: cross
Abstract: Training image captioning models using teacher forcing results in very generic samples, whereas more distinctive captions can be very useful in retrieval applications or to produce alternative texts describing images for accessibility. Reinforcement Learning (RL) allows to use cross-modal retrieval similarity score between the generated caption and the input image as reward to guide the training, leading to more distinctive captions. Recent studies show that pre-trained cross-modal retrieval models can be used to provide this …

abstract accessibility applications arxiv captioning captions clip cs.cl cs.cv image images modal reinforcement reinforcement learning retrieval samples training truth type