A Probabilistic Approach for Alignment with Human Comparisons

arXiv:2403.10771v1 Announce Type: new
Abstract: A growing trend involves integrating human knowledge into learning frameworks, leveraging subtle human feedback to refine AI models. Despite these advances, no comprehensive theoretical framework describing the specific conditions under which human comparisons improve the traditional supervised fine-tuning process has been developed. To bridge this gap, this paper studies the effective use of human comparisons to address limitations arising from noisy data and high-dimensional models. We propose a two-stage "Supervised Fine Tuning+Human Comparison" (SFT+HC) framework …

abstract advances ai models alignment arxiv bridge cs.lg feedback fine-tuning framework frameworks gap human human feedback knowledge process refine stat.ml supervised fine-tuning trend type