Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward

arXiv:2404.01258v1 Announce Type: new
Abstract: Preference modeling techniques, such as direct preference optimization (DPO), has shown effective in enhancing the generalization abilities of large language model (LLM). However, in tasks involving video instruction-following, providing informative feedback, especially for detecting hallucinations in generated responses, remains a significant challenge. Previous studies have explored using large large multimodal models (LMMs) as reward models to guide preference modeling, but their ability to accurately assess the factuality of generated responses compared to corresponding videos has …

abstract arxiv challenge cs.ai cs.cv direct preference optimization feedback generated hallucinations however language language model large language large language model large multimodal models llm modeling multimodal multimodal models optimization responses tasks type video