Low Variance Off-policy Evaluation with State-based Importance Sampling

arXiv:2212.03932v5 Announce Type: replace
Abstract: In many domains, the exploration process of reinforcement learning will be too costly as it requires trying out suboptimal policies, resulting in a need for off-policy evaluation, in which a target policy is evaluated based on data collected from a known behaviour policy. In this context, importance sampling estimators provide estimates for the expected return by weighting the trajectory based on the probability ratio of the target policy and the behaviour policy. Unfortunately, such estimators …

abstract arxiv context cs.ai cs.lg data domains evaluation exploration importance low policies policy process reinforcement reinforcement learning sampling state type variance will