Counterfactuals for Reinforcement Learning II: Improving Reward Learning

Safer reward function learning using counterfactuals

In the previous part of this series, I introduced counterfactuals and showed how to encode them in the POMDP framework. In this part, I will focus on how counterfactuals can be applied in the emerging field of Reward Learning. The article will first give a brief summary of the basic elements of Reward Learning. Using a running example, I will then demonstrate how Reward Learning can fail to produce the desired outcome. Ultimately, …

ai-alignment-and-safety ai-safety artificial intelligence editors pick ii learning reinforcement learning