Android in the Zoo: Chain-of-Action-Thought for GUI Agents

arXiv:2403.02713v1 Announce Type: cross
Abstract: Large language model (LLM) leads to a surge of autonomous GUI agents for smartphone, which completes a task triggered by natural language through predicting a sequence of actions of API. Even though the task highly relies on past actions and visual observations, existing studies typical consider little semantic information carried out by intermediate screenshots and screen operations. To address this, this work presents Chain-of-Action-Thought (dubbed CoAT), which takes the description of the previous actions, the …

abstract agents android api arxiv autonomous cs.cl cs.cv cs.hc cs.lg gui language language model large language large language model leads llm natural natural language smartphone studies thought through type visual