[D] [P] UI-based AI agents: UI-Act

Hi!

Happy to share a project I've been working on for a while: **UI-Act**

[**https://github.com/TobiasNorlund/UI-Act**](https://github.com/TobiasNorlund/UI-Act)

It's an AI model architecture designed to autonomously navigate and interact with computers using the graphical user interface. Think of it as a co-pilot that "sees" your screen and acts on it, just as a human would.

In essence, it's a custom transformer model taking prompt and screenshots as input, with output heads to predict low-level actions i.e. mouse clicks. In the demo, it has …

act agents ai agents ai model architecture computers co-pilot human machinelearning pilot project think