Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioning. (arXiv:2208.08831v1 [cs.CV])

Automatically discovering failures in vision models under real-world settings
remains an open challenge. This work demonstrates how off-the-shelf,
large-scale, image-to-text and text-to-image models, trained on vast amounts of
data, can be leveraged to automatically find such failures. In essence, a
conditional text-to-image generative model is used to generate large amounts of
synthetic, yet realistic, inputs given a ground-truth label. Misclassified
inputs are clustered and a captioning model is used to describe each cluster.
Each cluster's description is used in turn …

arxiv bugs captioning cv generation image image generation vision