Overview
agent-desktop is an open-source desktop control layer from Vercel Labs, version 1.0.2 under Apache-2.0. It does one specific thing: it lets a coding agent with shell access read and operate native applications through the operating system accessibility tree, instead of taking screenshots and asking a vision model to guess coordinates.
One command does one step. A snapshot returns a tree in which every actionable element carries a ref. The agent acts on that ref with commands such as click or type, then snapshots again to confirm the result. The loop stays in the agent's own code; the tool only observes and executes. A ref belongs to a particular snapshot, so when the interface changes it returns STALE_REF or AMBIGUOUS_TARGET and the agent looks again rather than clicking a row that has moved.
It ships as a Rust CLI plus a C library, installs from npm, and has native binaries for macOS, Windows and Linux. Execution is headless by default, without moving the pointer or stealing focus; physical input requires the --headed flag.
Key Features
- Accessibility tree snapshots: A snapshot returns JSON where each actionable element carries a role, a name and a ref, so the agent reads structure rather than pixels.
- Refs and failure codes: Refs are scoped to one snapshot and a bare @e8 is rejected. Interface changes return STALE_REF, and multiple windows return AMBIGUOUS_TARGET with the candidate window ids listed.
- Headless by default: Ref actions go through the accessibility API without taking focus or synthesising input. Real keyboard and mouse events require --headed.
- Three platforms: macOS needs accessibility permission, Windows needs no grant at the terminal's integrity level, and Linux needs a graphical session with at-spi2-core.
- Two install shapes: A global npm install brings native binaries and platform helpers, with archives per platform as an alternative. Official Homebrew and Scoop packages are still being submitted.
- CLI plus C library: The command line is usable by people and agents directly, while the C library supports bindings in other languages. Every command prints one JSON envelope, which is easy to parse and assert on.
Use Cases
- Automating legacy desktop software that has no API, locating buttons and fields by element semantics.
- Putting desktop steps inside a coding agent's loop: observe, act, observe again.
- Running desktop flows headless on CI or a server without windows popping up or focus changing.
- Batch file and form chores, locating entries in Finder or Explorer before operating on them.
- Regression checks for desktop apps, comparing interface structure after a sequence of clicks.
- Moving data between applications by reading in one window and writing in another.
Pros
- Element semantics survive theme changes and resolution shifts that would break coordinates.
- Explicit failure codes tell the agent to take a fresh snapshot instead of retrying blindly.
- Headless execution leaves the machine usable while a task runs.
- One command set across three platforms means scripts do not fork by system.
- Apache-2.0 licensing allows commercial use and modification.
- Structured JSON output is easy for an agent to parse, log and assert against.
Pricing
Free and open source under Apache-2.0. Install with npm install -g agent-desktop, or use the native archives for macOS, Windows and Linux; official Homebrew and Scoop packages are still being submitted. There is no hosted service and no usage metering.
Summary
Vision-based automation is brittle by nature. A new theme, a dialog, a different resolution, and the coordinates drift. agent-desktop takes the duller and steadier route, reading the accessibility tree the operating system already maintains, where elements have roles, names and stable refs. The cost is that it can only drive applications willing to expose accessibility information; the benefit is that every action can be verified. For anyone putting desktop work into CI or a background agent, headless execution and explicit failure codes matter more than multimodal recognition.
Version History
- agent-desktop 1.0.2: driving desktop apps through the accessibility tree (2026-11-01): Vercel Labs open-sourced agent-desktop 1.0.2 under Apache-2.0, a Rust CLI plus C library installed from npm and covering macOS, Windows and Linux. Snapshots come from the OS accessibility tree with a stable ref per actionable element, and the agent acts on a ref then snapshots again to verify, without screenshots or a vision model. Execution is headless by default with no focus stealing and no synthesised input; interface changes return STALE_REF and multiple windows return AMBIGUOUS_TARGET. macOS requires accessibility permission, Linux requires at-spi2-core, and Windows needs no grant at the terminal's integrity level.