Comments URL: https://news.ycombinator.com/item?id=48134594

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48017026

Points: 3

# Comments: 0

- Closing the coding feedback loop by having agents verify their own changes in a real app

- Automating repetitive workflows across apps that don't have good APIs

- Agents recording product demos of them using software. One compelling use case here: https://x.com/trycua/status/2047383207612645426

- Creating CLI and APIs for apps by reverse implementing their GUI, e.g. see: https://github.com/HKUDS/CLI-Anything

If you want to use it directly as an automation framework, you can take a Swift dependency on 'CuaDriverCore': https://cua.ai/docs/cua-driver/guide/getting-started/swift-i...

One useful trick is to cua-driver 'launch_app' instead of the default 'open' or other osascript, since it can start the app without raising/focusing it, and the tests don't disturb your active desktop while they run

The main problem: when a UI automation process controls a desktop app today, it usually takes over the human’s session. Your cursor moves, keyboard focus gets stolen, windows jump to the front, and you have to stop working until the agent is done. That is why we have historically avoided encouraging users to run these processes directly on their host machine, instead relying on VMs or GUI containers for concurrency and background execution.

But computer-use - the tools we give agents to operate computers like humans - does not scale cleanly that way. As models get smarter, agents need to share hosts safely, run in the background, and avoid collisions with the human or other agents using the same machine.

We realized macOS has no first-class API for "drive this app without touching the cursor". CGEventPost routes through the hardware input stream, so it moves your cursor. CGEvent.postToPid avoids the cursor warp, but Chromium treats those events as untrusted and silently drops clicks at the renderer boundary. Activating the target app first raises the window and pulls focus, defeating the point of background execution.

Cua Driver is our attempt at a real fix: a background computer-use driver for macOS that lets an agent click, type, scroll, and read native apps while your cursor, frontmost app, and Space stay where they are. The default interface is a CLI, so it is easy to script or call from any coding agent shell.

Try it on macOS 14+:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-d...)"

The first internal use case was delegated demo recording. We ask Claude Code to drive an app while 'cua-driver recording start' captures the trajectory, screenshots, actions, and click markers. The result is an agent-generated product demo, Screen Studio inspired.

Other things we have used it for:

- Replacing Vercel’s agent-browser and other browser-use CLIs. With Claude Code and Cua Driver, you do not need Chrome DevTools Protocol at all.

- A dev-loop QA agent that reproduces a visual bug, edits code, rebuilds, and verifies the UI while my editor stays frontmost.

- Personal-assistant flows that use iMessage from Claude Code, Hermes, or other general-purpose agent CLIs.

- Pulling visual context from Chrome, Figma, Preview, or YouTube windows I am not looking at, without relying on their APIs.

What made this harder than expected:

- CGEventPost warps the cursor because it goes through the HID stream.

- CGEvent.postToPid does not warp the cursor, but Chromium drops it at the renderer IPC boundary.

- Activating the target first raises the window and can drag you across Spaces.

- Electron apps stop keeping useful AX trees alive when windows are occluded without a private remote-aware SPI.

The unlock was SkyLight. SLEventPostToPid is a sibling of the public per-PID call, but it travels through a WindowServer channel Chromium accepts as trusted. Pair it with yabai’s focus-without-raise pattern, plus an off-screen primer click at (-1, -1), and the click lands without the window ever raising.

One thing we learned: the right addressing mode depends on the app. Native macOS apps usually have rich AX trees, Chromium-family apps often need a hybrid of AX and screenshots, and apps like Blender or CAD tools may expose almost no useful AX surface. The mistake is defaulting to pixels everywhere - or defaulting to AX everywhere.

Long technical writeup: https://github.com/trycua/cua/blob/main/blog/inside-macos-wi...

I would like feedback from people building Mac automation, agent harnesses, or accessibility tooling. If it breaks on an macOS app you care about, that is useful data for us.

Comments URL: https://news.ycombinator.com/item?id=47936312

Points: 192

# Comments: 43

Comments URL: https://news.ycombinator.com/item?id=47886802

Points: 1

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=47878715

Points: 4

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=47842545

Points: 2

# Comments: 0

What this is: CuaBot is a TUI to launch any CLI agent (Claude Code, OpenClaw, Codex, etc.) or GUI app inside a sandbox with computer-use. Agent windows appear natively on your desktop with a colored border. Your agent interacts with them through its own cursor, without any mouse/focus hijacking or invasive full-desktop screenshots.

This enables co-op mode: you and your agent work in the same windows with separate cursors.

What you can do:

$ npx cuabot claude > "Write a 2-player tic-tac-toe game, then let's play. I'll go first"

Claude Code will open the game in a sandboxed window on your desktop. When ready, you click your move through the native window while the agent watches and waits to click its move. The agent can see your cursor and its windows while keeping your full desktop isolated.

# Run agents in parallel:

$ npx cuabot -n research openclaw

$ npx cuabot -n coding codex

# Or script the CLI:

$ npx cuabot libreoffice --writer &

$ npx cuabot --click 150 48

$ npx cuabot --type “I Cua!”

Right now my cuabot agent is exploring mobile/desktop apps to turn into cuabench RL environments. I watch the windows appear, intervene when it gets stuck, and let it continue until it opens the completed GUI gym for me to interact with.

Why we built this:

We built Cua (https://github.com/trycua/cua), our open-source SDK for building and benchmarking computer-use systems with GUI sandboxes. We kept seeing two common patterns:

1. Agent screenshots your desktop and controls your mouse – Works with your data, but unsafe and locks you out

2. Agent runs in a sandbox with an external VNC desktop – Safer (our default), but clunky to monitor, hard to interact with, and tedious for data transfer

General computer-use should be frictionless. Asking your agent to debug a GUI app shouldn't require opening an entire desktop stream. The GUI app should just appear alongside your windows, sandboxed and ready.

How it works:

cuabot [command] launches cuabotd, which manages a Ubuntu + Xpra Docker container, a multi-cursor overlay, an Xpra computer-use MCP server, and an Xpra seamless client. It auto-configures your agent (Claude, Aider, etc.) to connect to the computer-use MCP, then pipes terminal I/O through WebSocket. The Xpra client automatically detects and streams windows launched in the container, with H.264 encoding, audio, and customizable clipboard sharing.

Since the computer-use MCP interacts through an Xpra client, the agent only sees the windows it needs, sparing it from your desktop clutter!

GitHub: https://github.com/trycua/cua (monorepo; libs/cuabot directory)

Docs: https://cua.ai/docs

npm: https://www.npmjs.com/package/cuabot

We'll be here to answer questions!

Comments URL: https://news.ycombinator.com/item?id=46887532

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=46844214

Points: 4

# Comments: 2

Comments URL: https://news.ycombinator.com/item?id=46828620

Points: 2

# Comments: 0