Tackling Structural Hallucination in Image Translation with Local Diffusion

arXiv:2404.05980v1 Announce Type: cross
Abstract: Recent developments in diffusion models have advanced conditioned image generation, yet they struggle with reconstructing out-of-distribution (OOD) images, such as unseen tumors in medical images, causing ``image hallucination'' and risking misdiagnosis. We hypothesize such hallucinations result from local OOD regions in the conditional images. We verify that partitioning the OOD region and conducting separate image generations alleviates hallucinations in several applications. From this, we propose a training-free diffusion framework that reduces hallucination with multiple Local …

abstract advanced arxiv cs.ai cs.cv cs.lg diffusion diffusion models distribution hallucination hallucinations image image generation images medical struggle translation tumors type verify