Self-Supervised Visual Preference Alignment

arXiv:2404.10501v1 Announce Type: cross
Abstract: This paper makes the first attempt towards unsupervised preference alignment in Vision-Language Models (VLMs). We generate chosen and rejected responses with regard to the original and augmented image pairs, and conduct preference alignment with direct preference optimization. It is based on a core idea: properly designed augmentation to the image input will induce VLM to generate false but hard negative responses, which helps the model to learn from and produce more robust and powerful answers. …

abstract alignment arxiv augmentation core cs.ai cs.cl cs.cv cs.lg direct preference optimization generate image language language models optimization paper regard responses type unsupervised vision vision-language vision-language models visual vlms