[R] Towards General Computer Control: A Multimodal Agent for Red Dead Redemption II as a Case Study - Beijing Academy of Artificial Intelligence (BAAI) 2024 - First Agent able to follow and finish real missions in a AAA game!

Paper: [https://arxiv.org/abs/2403.03186](https://arxiv.org/abs/2403.03186)

Projekt Website with code and videos: [https://baai-agents.github.io/Cradle/](https://baai-agents.github.io/Cradle/)

Abstract:

>Despite the success in specific tasks and scenarios, existing foundation agents, empowered by large models (LMs) and advanced tools, still cannot generalize to different scenarios, mainly due to dramatic differences in the observations and actions across scenarios. In this work, we propose the General Computer Control (GCC) setting: building foundation agents that can master any computer task by taking only screen images (and possibly audio) of the computer as input, …

abstract advanced agents building computer control differences foundation gcc general large models lms machinelearning master specific tasks success tasks tools work