Risk-Sensitive RL with Optimized Certainty Equivalents via Reduction to Standard RL

arXiv:2403.06323v1 Announce Type: new
Abstract: We study Risk-Sensitive Reinforcement Learning (RSRL) with the Optimized Certainty Equivalent (OCE) risk, which generalizes Conditional Value-at-risk (CVaR), entropic risk and Markowitz's mean-variance. Using an augmented Markov Decision Process (MDP), we propose two general meta-algorithms via reductions to standard RL: one based on optimistic algorithms and another based on policy optimization. Our optimistic meta-algorithm generalizes almost all prior RSRL theory with entropic risk or CVaR. Under discrete rewards, our optimistic theory also certifies the first …

abstract algorithms arxiv cs.lg decision general markov mean meta process reinforcement reinforcement learning risk standard study type value variance via