No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO

arXiv:2405.00662v1 Announce Type: new
Abstract: Reinforcement learning (RL) is inherently rife with non-stationarity since the states and rewards the agent observes during training depend on its changing policy. Therefore, networks in deep RL must be capable of adapting to new observations and fitting new targets. However, previous works have observed that networks in off-policy deep value-based methods exhibit a decrease in representation rank, often correlated with an inability to continue learning or a collapse in performance. Although this phenomenon has …

abstract agent arxiv cs.lg deep rl however networks policy ppo reinforcement reinforcement learning representation targets training trust trust issues type