Revisiting Recurrent Reinforcement Learning with Memory Monoids

arXiv:2402.09900v1 Announce Type: new
Abstract: In RL, memory models such as RNNs and transformers address Partially Observable Markov Decision Processes (POMDPs) by mapping trajectories to latent Markov states. Neither model scales particularly well to long sequences, especially compared to an emerging class of memory models sometimes called linear recurrent models. We discover that the recurrent update of these models is a monoid, leading us to formally define a novel memory monoid framework. We revisit the traditional approach to batching in …

abstract arxiv class cs.ai cs.lg decision linear mapping markov memory observable processes reinforcement reinforcement learning transformers type