Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Models

arXiv:2404.01863v1 Announce Type: new
Abstract: Fine-tuning text-to-image models with reward functions trained on human feedback data has proven effective for aligning model behavior with human intent. However, excessive optimization with such reward models, which serve as mere proxy objectives, can compromise the performance of fine-tuned models, a phenomenon known as reward overoptimization. To investigate this issue in depth, we introduce the Text-Image Alignment Assessment (TIA2) benchmark, which comprises a diverse collection of text prompts, images, and human annotations. Our evaluation …

abstract arxiv behavior confidence cs.ai cs.lg data feedback fine-tuning functions however human human feedback image model behavior optimization performance serve text text-to-image type