OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web

arXiv:2402.17553v1 Announce Type: cross
Abstract: For decades, human-computer interaction has fundamentally been manual. Even today, almost all productive work done on the computer necessitates human input at every step. Autonomous virtual agents represent an exciting step in automating many of these menial tasks. Virtual agents would empower users with limited technical proficiency to harness the full possibilities of computer systems. They could also enable the efficient streamlining of numerous computer tasks, ranging from calendar management to complex travel bookings, with …

abstract agents arxiv autonomous autonomous agents benchmark computer cs.ai cs.cl cs.cv cs.hc dataset desktop enabling every human human-computer interaction multimodal productive tasks type virtual virtual agents web work