Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model

arXiv:2311.13231v3 Announce Type: replace
Abstract: Using reinforcement learning with human feedback (RLHF) has shown significant promise in fine-tuning diffusion models. Previous methods start by training a reward model that aligns with human preferences, then leverage RL techniques to fine-tune the underlying models. However, crafting an efficient reward model demands extensive datasets, optimal architecture, and manual hyperparameter tuning, making the process both time and cost-intensive. The direct preference optimization (DPO) method, effective in fine-tuning large language models, eliminates the necessity for …

arxiv cs.ai cs.cv cs.lg diffusion diffusion models feedback human human feedback reward model type