Q-Probe: A Lightweight Approach to Reward Maximization for Language Models

arXiv:2402.14688v1 Announce Type: new
Abstract: We present an approach called Q-probing to adapt a pre-trained language model to maximize a task-specific reward function. At a high level, Q-probing sits between heavier approaches such as finetuning and lighter approaches such as few shot prompting, but can also be combined with either. The idea is to learn a simple linear function on a model's embedding space that can be used to reweight candidate completions. We theoretically show that this sampling procedure is …

arxiv cs.lg language language models probe type