Data Alignment for Zero-Shot Concept Generation in Dermatology AI

arXiv:2404.13043v1 Announce Type: cross
Abstract: AI in dermatology is evolving at a rapid pace but the major limitation to training trustworthy classifiers is the scarcity of data with ground-truth concept level labels, which are meta-labels semantically meaningful to humans. Foundation models like CLIP providing zero-shot capabilities can help alleviate this challenge by leveraging vast amounts of image-caption pairs available on the internet. CLIP can be fine-tuned using domain specific image-caption pairs to improve classification performance. However, CLIP's pre-training data is …

abstract alignment arxiv capabilities challenge classifiers clip concept cs.cl cs.cv cs.lg data dermatology foundation ground-truth humans labels major meta training trustworthy truth type zero-shot