[D] what's the proper way of doing direct preference optimization (DPO) and why?

For some reason I just could not wrap my mind around the data distribution problem with DPO. In the paper it says:

https://preview.redd.it/6c9z61o4bbfc1.png?width=2164&format=png&auto=webp&s=c6b5ed46937da04e5912023e2f46ae7821a9a446

My question is: why does it matter so much that the preference data distribution aligns with the reference model output distribution? My understanding is that during training, the parameters of the sft are updated such that chosen responses (y\_w) have a higher probability of being generated, and rejected responses (y\_l) have a lower probability of being generated, …

data direct preference optimization distribution machinelearning matter mind optimization paper question reason reference understanding