Exposing Text-Image Inconsistency Using Diffusion Models

arXiv:2404.18033v1 Announce Type: new
Abstract: In the battle against widespread online misinformation, a growing problem is text-image inconsistency, where images are misleadingly paired with texts with different intent or meaning. Existing classification-based methods for text-image inconsistency can identify contextual inconsistencies but fail to provide explainable justifications for their decisions that humans can understand. Although more nuanced, human evaluation is impractical at scale and susceptible to errors. To address these limitations, this study introduces D-TIIL (Diffusion-based Text-Image Inconsistency Localization), which employs …

abstract arxiv classification cs.cv decisions diffusion diffusion models humans identify image images meaning misinformation text text-image type