Mitigating Open-Vocabulary Caption Hallucinations

arXiv:2312.03631v2 Announce Type: replace
Abstract: While recent years have seen rapid progress in image-conditioned text generation, image captioning still suffers from the fundamental issue of hallucinations, namely, the generation of spurious details that cannot be inferred from the given image. Existing methods largely use closed-vocabulary object lists to mitigate or evaluate hallucinations in image captioning, ignoring most types of hallucinations that occur in practice. To this end, we propose a framework for addressing hallucinations in image captioning in the open-vocabulary …

abstract arxiv captioning cs.ai cs.cv hallucinations image issue lists progress text text generation type