Agent Desktop

Agent Desktop is an open-source tool that gives AI agents reliable computer use on macOS. Published on GitHub by lahfir under the Apache-2.0 license, it is built in Rust and works through the operating system’s accessibility trees instead of screenshots: the agent sees each app’s real UI structure, element references stay stable across steps, and actions are safe to retry rather than guessed from pixels. The project has around 1,800 stars and is actively developed, with code last updated in 2026.

Overview

Pixel-based computer use is brittle: a window moves, a layout shifts, and the agent loses track of what it was clicking. Agent Desktop takes the structural route, exposing the accessibility tree of any app so agents can address real interface elements by reference. It ships as a CLI and speaks MCP, so agent frameworks and coding assistants can plug desktop control into their tool layer without custom screenshot pipelines.

Key Features

• Accessibility-tree based perception: agents read real UI structure instead of guessing from pixels
• Stable element references that survive layout changes and make actions safe to retry
• Works with any app that exposes macOS accessibility information
• MCP support for direct integration with agent frameworks and AI coding tools
• Rust implementation with a CLI interface for scripting and automation
• Apache-2.0 licensed and under active development

💳 Pricing

Free. Agent Desktop is open source under the Apache-2.0 license; you download and run it locally at no cost.

Who Is It For

Developers building computer-use agents, agent framework authors who need dependable desktop control, and automation engineers tired of fragile screenshot-and-click scripts on macOS.