UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction

Abstract

Developing autonomous agents that can navigate diverse Graphical User Interfaces (GUIs) and solve complex tasks is essential for understanding and accelerating human workflows. Currently, most top-performing models and benchmarks primarily focus on either web or mobile platforms, while desktop platforms – equally prevalent in user interfaces – are often overlooked due to licensing issues and the high cost of data collection. The complexity and variability of desktop GUIs present a major challenge in automation, making high-quality datasets an important step toward addressing these capabilities. In this work, we introduce UI-Vision, a comprehensive desktop-centric and license-permissive GUI understanding benchmark designed to tackle these challenges. UI-Vision features: (i) high-quality action trajectories recorded through human demonstrations and refined via expert human annotators; (ii) a wide range of annotations, including bounding boxes, action trajectories, and layout information; and (iii) significant diversity, spanning 83 software applications. Our evaluation reveals significant limitations in the ability of current models to effectively handle desktop environments, highlighting the considerable progress needed to achieve truly autonomous GUI agents capable of generalizing from visual cues. To support the broader research community, all our data and resources will be made fully open-source.