Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis

arXiv:2404.08003v1 Announce Type: new
Abstract: To improve the efficiency of reinforcement learning, we propose a novel asynchronous federated reinforcement learning framework termed AFedPG, which constructs a global model through collaboration among $N$ agents using policy gradient (PG) updates. To handle the challenge of lagged policies in asynchronous settings, we design delay-adaptive lookahead and normalized update techniques that can effectively handle the heterogeneous arrival times of policy gradients. We analyze the theoretical global convergence bound of AFedPG, and characterize the advantage …

abstract agents algorithm algorithm design analysis arxiv asynchronous challenge collaboration convergence cs.dc cs.lg cs.ni design efficiency framework global gradient novel policies policy reinforcement reinforcement learning through type updates