GUing: A Mobile GUI Search Engine using a Vision-Language Model

arXiv:2405.00145v1 Announce Type: cross
Abstract: App developers use the Graphical User Interface (GUI) of other apps as an important source of inspiration to design and improve their own apps. In recent years, research suggested various approaches to retrieve GUI designs that fit a certain text query from screenshot datasets acquired through automated GUI exploration. However, such text-to-GUI retrieval approaches only leverage the textual information of the GUI elements in the screenshots, neglecting visual information such as icons or background images. …

abstract acquired app apps arxiv cs.cv cs.se datasets design designs developers gui inspiration language language model mobile query research search search engine text type vision vision-language