Preventing Reward Hacking with Occupancy Measure Regularization

arXiv:2403.03185v1 Announce Type: new
Abstract: Reward hacking occurs when an agent performs very well with respect to a "proxy" reward function (which may be hand-specified or learned), but poorly with respect to the unknown true reward. Since ensuring good alignment between the proxy and true reward is extremely difficult, one approach to prevent reward hacking is optimizing the proxy conservatively. Prior work has particularly focused on enforcing the learned policy to behave similarly to a "safe" policy by penalizing the …

arxiv cs.ai cs.lg hacking regularization type