On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluation

arXiv:2402.14703v1 Announce Type: new
Abstract: We study off-policy evaluation (OPE) in partially observable environments with complex observations, with the goal of developing estimators whose guarantee avoids exponential dependence on the horizon. While such estimators exist for MDPs and POMDPs can be converted to history-based MDPs, their estimation errors depend on the state-density ratio for MDPs which becomes history ratios after conversion, an exponential object. Recently, Uehara et al. (2022) proposed future-dependent value functions as a promising framework to address this …

abstract arxiv cs.ai cs.lg environments evaluation functions future history horizon observable policy stat.ml study type value