Tell Me What's Next: Textual Foresight for Generic UI Representations

arXiv:2406.07822v1 Announce Type: cross
Abstract: Mobile app user interfaces (UIs) are rich with action, text, structure, and image content that can be utilized to learn generic UI representations for tasks like automating user commands, summarizing content, and evaluating the accessibility of user interfaces. Prior work has learned strong visual representations with local or global captioning losses, but fails to retain both granularities. To combat this, we propose Textual Foresight, a novel pretraining objective for learning UI screen representations. Textual Foresight …

abstract accessibility action app arxiv cs.cl cs.cv image interfaces learn mobile mobile app next prior summarizing tasks text textual type visual work