stupidmirror-device-use
Observe and operate real iPhone or Android devices using StupidMirror tools, including reading screens, app navigation, taps, swipes, text entry and real-device UI checks. Use when the user asks to operate a connected phone; connection setup uses stupidmirror-setup.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 1.6k
- Forks
- 491
01 前言 Front Matter
- name
- stupidmirror-device-use
- description
- Observe and operate real iPhone or Android devices using StupidMirror tools, including reading screens, app navigation, taps, swipes, text entry and real-device UI checks. Use when the user asks to operate a connected phone; connection setup uses stupidmirror-setup.
03 SKILL.md 正文
Operate a device through StupidMirror
Use the StupidMirror MCP tools. Start with list_devices; select the intended device and consistently pass device_id when multiple devices exist. Call show_screen to display the live device inside Codex, then connect_control if actions are needed. The app captures in the background. Do not call start_mirror unless the user explicitly wants a separate Mac mirror window.
- When the intended target is known, call
tap_textdirectly with candidate labels inqueries; avoid a separate read/find/tap chain. Usefind_any_elementwhen you need to inspect or disambiguate a target before tapping. These share one local OCR pass and at most one native hierarchy fallback; do not query synonyms separately. observe_screendefaults to an image plus compact OCR, without a native hierarchy. Keep the image for visual tasks or when the needed information is unclear; for clearly text-only reads, explicitly setinclude_image=false.observe_screen,find_elementandfind_any_elementdefault toresponse_format="compact". Keep that format for routine work; do not request full output and then filter it in model context.- Compact elements retain
id, text andsource;center: [x, y]is a rounded normalized hint for distinguishing repeated labels. Native elements also retain type and available state flags/values. Use the elementidand observationid(or searchobservationID) aselement_idandobservation_idfortap_element: it uses the stored full-precision geometry. Reobserve after navigation; do not reuse stale ids or treat OCR text as proof of clickability or toggle state. - To verify an expected text, use
assert_screenor a boundedwait_forinstead of reading the whole screen. A successful tap alone does not prove the destination appeared. - Request
include_accessibility=true, include_ocr=falsefor native switch values, selection, enabled state or semantics absent from OCR. Do not fetch a native tree on every read. Useresponse_format="full"only when raw bounds, confidence precision or parent/child hierarchy are needed for diagnostics; it does not itself enable images or accessibility. - Keep
include_image=true(the default) for unlabeled icons, images, maps, canvas content, layout checks or unresolved ambiguity. Inspect the fresh image before coordinate input. If only the image is needed, also passinclude_ocr=false. Do not take a screenshot automatically after every action. - Once a field is focused, call
replace_textto overwrite,clear_textto empty, ortype_textto append. Do not fetch an accessibility tree between focus and input: an iOS snapshot can disturb focus. Verify the resulting text before submission. highlight_elementsandhighlight_clickable_elementsannotate a separately opened Mac mirror without tapping the phone; these native overlays are not currently drawn in the embedded preview.
If an action loses its connection or times out, it may already have executed. Read the current state before deciding what to do next; do not blindly repeat clicks, text entry or submission. Use diagnostics for persistent failures and reuse the app's device connection.
Only carry out the user's requested actions. Drafting text does not authorize sending it, making a purchase or publishing content. Leave authentication secrets and biometric prompts to the user.
When the user sends a screenshot annotation, use its device ID and note. The attached image and numbered regions describe a frozen frame; observe the live device again before an action, and confirm the intended target still matches. Do not blindly replay the stored coordinates. Preview uses WebRTC independently of AI tool calls. screen_signal, screen_frame and screen_action are app-only transport/UI tools.
04 同仓库更多 Skills
worktable-review-with-annotations
Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.
worktable-find-and-synthesize-context
Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.
worktable-create-or-update-docs
Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.
worktable-create-or-manage-records
Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.
worktable-create-or-manage-interactive-html
Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.
worktable-collaborate-threads
Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.