Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles

arXiv:2303.03751v2 Announce Type: replace
Abstract: In this study, we delve into an emerging optimization challenge involving a black-box objective function that can only be gauged via a ranking oracle-a situation frequently encountered in real-world scenarios, especially when the function is evaluated by human judges. Such challenge is inspired from Reinforcement Learning with Human Feedback (RLHF), an approach recently employed to enhance the performance of Large Language Models (LLMs) using human guidance. We introduce ZO-RankSGD, an innovative zeroth-order optimization algorithm designed …

abstract arxiv box challenge cs.ai cs.lg feedback function human human feedback judges optimization oracle ranking reinforcement study type via world