computer-use
Use when a task needs a native desktop app's own UI or the OS. For anything inside a web page, use Browser Use. Main agent only.
- License 许可
- Apache-2.0
- 所属仓库
- zai-org/ZCode
- 最近更新
- 2026-11-13
- Stars
- ★ 8.7k
- Forks
- 2.7k
01 前言 Front Matter
- name
- computer-use
- description
- Use when a task needs a native desktop app's own UI or the OS. For anything inside a web page, use Browser Use. Main agent only.
03 SKILL.md 正文
Computer Use
Read or operate the UI of native apps on the user's computer.
- Prefer a dedicated connector, API, CLI or skill when one can complete the task.
- For browser and web tasks, use Browser Use instead.
- Do not use AppleScript,
osascript, JXA, System Events, shell commands, or any other UI-automation technology unless the user explicitly asks for that technology. - Main agent only. Never delegate Computer Use to a subagent.
Bootstrap every call
Each mcp__node_repl__js call runs in a fresh Worker. Globals, imports, module
cache and any binding are gone by the next call, so const app does not survive
the end of the cell. The UI state does survive, so re-binding in the next cell is
cheap.
Put the bootstrap and the actions in the same cell, bootstrap first:
const root =
process.env.ZCODE_CUA_PLUGIN_ROOT ??
process.env.ZCODE_PLUGIN_ROOT ??
process.env.CLAUDE_PLUGIN_ROOT;
const { join } = await import("node:path");
const { pathToFileURL } = await import("node:url");
const { setupComputerUseRuntime } = await import(
pathToFileURL(join(root, "scripts", "computer-use-client.mjs")).href,
);
await setupComputerUseRuntime({ globals: globalThis });
Accessibility first
Accessibility is the primary action path. It is semantic, precise, works on a background app and does not steal the user's focus.
- Observe, then search the returned tree for the target by its role, name, title, value or other visible identity.
- When a matching element exists, act on it by index. For a control that
advertises a semantic action,
performSecondaryActionis also correct. - For a settable element prefer
setValueover typing or pasting. Reach forpasteonly when the target is not settable or the content is rich text. - Keyboard input is the fallback: use
pressKeyonly when no element expresses the operation, or the user asked for keyboard interaction. Coordinates are the last fallback, for canvas, games and Electron content accessibility cannot see.
Do not replace an available element action with a keyboard shortcut just because the shortcut is shorter. Do not run both the accessibility and visual paths for the same action.
Observation works on a background app, screenshots included — on any
display, including windows at negative coordinates.
has_image: false means the call did not ask for pixels (getAXState), never
that capture failed.
Success means the API accepted an action, not that the app acted. Typing into a web-content editor can be accepted and change nothing, so re-observe to confirm the text landed.
API
type Vec2 = [x: number, y: number];
type ObservationOptions = { emit?: boolean };
type StateOptions = ObservationOptions & { disableDiffing?: boolean };
type StateAndScreenshot = { state: string; screenshot?: Uint8Array };
type Direction = "up" | "down" | "left" | "right" | "u" | "d" | "l" | "r";
type MouseButton = "left" | "right" | "middle" | "l" | "r" | "m";
type SelectionType = "text" | "cursor_before" | "cursor_after";
type Strategy = "auto" | "a11y" | "event";
type ClickOptions = {
mouseButton?: MouseButton;
clickCount?: number;
modifiers?: string;
strategy?: Strategy;
};
type SelectTextOptions = { prefix?: string; suffix?: string; selectionType?: SelectionType };
type PasteOptions = { format?: "text" | "md" | "html" };
type PressKeyOptions = { holdSeconds?: number; strategy?: Strategy };
interface Target {
getAXState(options?: StateOptions): Promise<string>;
getScreenshot(options?: ObservationOptions): Promise<Uint8Array>;
getAXStateAndScreenshot(options?: StateOptions): Promise<StateAndScreenshot>;
elements(): Promise<{ index: number; kind: string; title: string | null; value: string | null; actions: string[] }[]>;
paste(text: string, options?: PasteOptions): Promise<void>;
click(target: number | Vec2, options?: ClickOptions): Promise<void>;
drag(from: number | Vec2, to: number | Vec2, options?: { modifiers?: string }): Promise<void>;
pressKey(key: string, options?: PressKeyOptions): Promise<void>;
scroll(target: number | Vec2, direction: Direction, pages?: number): Promise<void>;
selectText(elementIndex: number, text: string, options?: SelectTextOptions): Promise<void>;
setValue(elementIndex: number, value: string): Promise<void>;
typeText(text: string): Promise<void>;
performSecondaryAction(elementIndex: number, action: string): Promise<void>;
}
interface App extends Target {}
type AppRef = { name?: string; bundle_id?: string; pid?: number; window_id?: number };
type AppInfo = { pid: number; name: string | null; bundle_id: string | null; active: boolean };
type State = { apps: AppInfo[] };
declare const agent: {
computerUse: {
getState(options?: ObservationOptions): Promise<State>;
getApp(target: string | AppRef): Promise<App>;
listApps(options?: ObservationOptions): Promise<AppInfo[]>;
computer: Record<string, (args: object) => Promise<unknown>>;
requestAccess(capabilities?: string[]): Promise<Record<string, unknown>>;
stop(reason?: string): Promise<void>;
};
};
The full reference is agent.documentation.get("computer-use"), fetched on demand;
read it only for an argument shape or response field this page does not give.
The loop
Observe once, act, then observe again before deciding the next step.
agent.computerUse.getApp(...) binds an app and shows nothing; call getAXState
when you need to see its state. Binding also launches it if not running; there is
no separate launch tool. Its argument is a display name or a bundle identifier —
the same strings listApps() returns — or an AppRef.
When the user names an application, copy it character-for-character into the app
identifier. Do not translate, localize, normalize, shorten, or remove a suffix:
{"name":"网易云音乐app"} is not {"name":"网易云音乐"}; {"name":"日历"} is not
{"name":"Calendar"}. A rewritten name resolves to a different app or to nothing,
and the failure reads "app not found". If the exact string does not resolve, call
agent.computerUse.listApps() once and pick the matching identifier.
Batch related actions and a single closing observation into one cell:
const app = await agent.computerUse.getApp("Notes");
await app.click(box);
await app.typeText("hello");
await app.pressKey("Return");
await app.getAXState();
An element index addresses that app's latest observation, so several actions may
reuse one index without re-observing between them — click an index, then type into
it, in the same cell. Observing renumbers the tree, so take indices from the newest
one. A vanished element fails closed with ELEMENT_UNAVAILABLE.
The tree comes back as a diff only against a tree this cell already showed you,
listing the elements that were removed, added or changed; unchanged rows are
omitted and their indices stay valid. The first tree after binding, and the first
after getScreenshot or elements(), are always complete. Pass
{ disableDiffing: true } for a full tree at any point. If a standalone
observation reports no change, do not immediately repeat it without an
intervening action.
A large tree is trimmed by priority (ancestors kept), the header says so, and
indices then skip numbers. app.elements() returns every element with its index,
trimmed ones included — filter it in JS, never guess an index.
A capture is scoped to one window: without a window_id the main/key window is
re-resolved every observation, so a modal that just opened becomes the captured
window. When an action fails or the tree reads like another part of the app, check
the observation's window — a dialog shows up there. Read it, then act on it, or
dismiss it (Escape or its own cancel) and observe again. A missing element is not
proof the action worked. Pin one with getApp({pid, window_id}); list_windows
has the id.
Output
Observations display themselves: getAXState, getScreenshot,
getAXStateAndScreenshot, getState and listApps emit their own result.
Never pass their return value to nodeRepl.write(...) or
nodeRepl.emitImage(...): a second raster in one result breaks the one-raster
rule and the frame is removed entirely, so you end up with no picture at all.
Pass { emit: false } to suppress the display and still receive the value.
Action methods display nothing.
Coordinates
Choose x and y only by looking at the current returned raster, with
0 <= x < width and 0 <= y < height for that raster. Submit those integers
unchanged; CUA binds the current raster internally and owns every transform from
the returned raster to native dispatch. Element and window bounds are diagnostic
global screen points and must never be copied into a coordinate. When visual
fallback begins, discard coordinate-like numbers from earlier text or
accessibility results.
Act on the current raster when the target is clear; if it is too small or ambiguous, re-observe rather than guessing at geometry.
A pointer action accepted with no change usually means the app acted where the real pointer sits: repeating it will not help — use an element index or the keyboard.
A coordinate refusal may name an unexpected owner, or say the frame is stale — the window moved, resized or was replaced. Observe again for a current raster, or act on an element index, which does not depend on window geometry. Never move, resize or close a window to make a coordinate land.
Keyboard
pressKey takes a key or a +-separated chord and accepts both short names and
X keysym style: "a", "Return", "Tab", "Control_L+a", "super+c", "Up".
macOS uses cmd; Linux and Windows use ctrl. Use { holdSeconds } to hold a
key or chord for a duration rather than simulating repeated presses.
Bind keyboard input to an app or element; never send it unbound. paste is for
rich text, or a target setValue cannot set — not for plain text a settable
element accepts.
performSecondaryAction accepts only an action the element advertises in the
current tree. Do not guess an action name.
At most one element is focused — the one holding keyboard focus; absent means
undetermined.
selectText locates text inside an editable element. Use prefix/suffix to
disambiguate repeated matches and selectionType to place the cursor instead of
selecting. An ambiguous match is refused rather than resolved to the first hit.
Waiting
Observations wait for the UI to settle before capturing. Do not pause or delay
before reading state — no setTimeout, no polling loop.
Errors and stopping
Actions resolve to undefined on success and throw ComputerUseError otherwise:
code—PERMISSION_DENIED,NOT_AUTHORIZED,APP_NOT_FOUND,AMBIGUOUS_APP,LAUNCH_FAILED,INVALID_APP,ELEMENT_UNAVAILABLE,STALE_STATE,NOT_SETTABLE,NOT_SELECTABLE,ACTION_UNAVAILABLE,FOREGROUND_REQUIRED,CONTROLLER_BUSY,CONTROL_STOPPED,SCREEN_LOCKED,HELPER_UNAVAILABLE,VERSION_MISMATCH,TIMEOUT,STRUCTURED_STATE_UNAVAILABLE,INTERNAL.actionSent— whether the action may already have reached the app. Retry a non-idempotent action only when this isfalse; otherwise observe first and decide from what you see.retry—"reobserve","retry"or"never".
CONTROLLER_BUSY means another live ZCode Computer Use session owns input. It is
never retryable: report the owner from the error and ask the user to close that
session.
Stop immediately after agent.computerUse.stop(), a kill switch, a permission
refusal, or a non-retryable error. Do not switch to a different UI-automation
technology after an access refusal.
Persist until the request is actually complete. Attempting an action is not completion: verify the returned state visibly shows the result. If it is unchanged or only intermediate, try another approach. Respond only when the requested state is visibly present, or explain a concrete blocker you cannot resolve.
Tool surface
agent.computerUse.computer.<tool>(args) is the low-level surface. Prefer the
bound-object API above; reach for a tool only for what the API does not
express — window enumeration, key repeat, or reading state back in the same
call. Arguments are strict: an undeclared key is refused. Each tool takes one
arguments object — the names below are its keys, not positional parameters:
get_app_state({ app_ref: { bundle_id: "com.apple.Notes" }, include_screenshot: true }).
Name an app as the OS lists it (Windows: the Start-menu name, not a window title).
Nothing takes the user's focus, except on Windows: launching a non-packaged app
does, and include_screenshot=true un-minimizes — a minimized tree is fully
usable, so keep the default.
list_apps({})
list_windows({app_ref})
get_app_state({app_ref, include_screenshot?=false, disable_diffing?=false})
left_click({target, mouse_button?="left", click_count?=1, modifiers?="",
strategy?, app_ref?, return_state?})
left_click_drag({from_target, to, modifiers?="", strategy?, app_ref?,
return_state?})
scroll({target, scroll_direction, scroll_amount, strategy?, app_ref?,
return_state?})
type({text, target?, app_ref?, strategy?, return_state?})
set_value({target, value, strategy?, app_ref?, return_state?})
select_text({target, text_range?, app_ref?, return_state?})
key({text, repeat?, hold_seconds?, app_ref?, strategy?, return_state?})
paste({text, format?="text", app_ref?, return_state?})
perform_action({target, action, app_ref?, return_state?})
request_access({capabilities?})
stop_computer_control({reason?})
Argument shapes: app_ref / app is an AppRef, but a bare string here is
read as a bundle id, so pass {name: "Notes"} for a display name.
scroll_direction is up|down|left|right, scroll_amount is pages, strategy
is auto|a11y|event, and return_state is compact|full|none to return the app
state in the same call. For target, text_range, modifiers and the response
shapes, see nodeRepl.write(await agent.documentation.get("computer-use")).
04 同仓库更多 Skills
plugin-mcp-ping
E2E workspace Plugin skill marker
e2e-ext-compat-plain
PLM-EXT-STORE fixture skill for e2e-ext-compat-plain.
e2e-ext-compat-ok
PLM-EXT-STORE fixture skill for e2e-ext-compat-ok.
e2e-ext-compat-installed
PLM-EXT-STORE fixture skill for e2e-ext-compat-installed.
e2e-ext-compat-future
PLM-EXT-STORE fixture skill for e2e-ext-compat-future.
e2e-store-skill-b
E2E store fixture skill b