ci-adhoc-test
Run ad-hoc / exploratory tests on a real OS or platform via CI when the behavior CANNOT be reproduced on the local host or in Docker — macOS Seatbelt / sandbox-exec / codesigning, Windows cmd.exe / --script-shell shell selection / .cmd resolution / Authenticode, musl-vs-glibc, Linux-arm64, a specific Node floor. Invoke (via the Skill tool) whenever the user asks to "test this on macOS/Windows CI", "probe this across operating systems / platforms", "run an ad-hoc cross-platform check", or any validation that needs a real macOS or Windows runner. THE KEY FACT this skill exists to carry: a pull request is NOT required — a branch-scoped GitHub Actions workflow (workflow_dispatch + push to the branch) runs the probe with no PR open. Pairs with `ad-hoc-test` (local host probing), `dev-loop` (build), `ci-watch` (await the run), and AGENTS.md's Docker section (Linux-only; this skill covers what Docker can't).
- License 许可
- MIT
- 所属仓库
- nubjs/nub
- 最近更新
- 2026-11-14
- Stars
- ★ 4.4k
- Forks
- 71
01 前言 Front Matter
- name
- ci-adhoc-test
- metadata
- { "internal": true }
- description
- Run ad-hoc / exploratory tests on a real OS or platform via CI when the behavior CANNOT be reproduced on the local host or in Docker — macOS Seatbelt / sandbox-exec / codesigning, Windows cmd.exe / --script-shell shell selection / .cmd resolution / Authenticode, musl-vs-glibc, Linux-arm64, a specific Node floor. Invoke (via the Skill tool) whenever the user asks to "test this on macOS/Windows CI", "probe this across operating systems / platforms", "run an ad-hoc cross-platform check", or any validation that needs a real macOS or Windows runner. THE KEY FACT this skill exists to carry: a pull request is NOT required — a branch-scoped GitHub Actions workflow (workflow_dispatch + push to the branch) runs the probe with no PR open. Pairs with `ad-hoc-test` (local host probing), `dev-loop` (build), `ci-watch` (await the run), and AGENTS.md's Docker section (Linux-only; this skill covers what Docker can't).
03 SKILL.md 正文
Ad-hoc testing on a real OS/platform via CI
Some behavior is only observable on a real target platform: macOS Seatbelt / sandbox-exec / Gatekeeper / codesigning, Windows cmd.exe / --script-shell selection / .cmd/.bat resolution / Authenticode / SmartScreen, musl-vs-glibc detection, Linux-arm64, a pinned Node floor. Docker closes the Linux corners cheaply but runs Linux containers only — it is not a substitute for macOS or Windows.
The key fact: no PR is required
A GitHub Actions workflow does not need an open pull request. Trigger it on the branch:
on:
push:
branches: [<your-probe-branch>] # THE trigger: runs on every push to THIS branch
paths: # only when the harness itself changes
- 'tests/<probe-name>/**'
- '.github/workflows/<wf>.yml'
workflow_dispatch: # INERT until the file lands on the default branch — see below
- Push to the branch → the probe runs. Open no PR, or close one you opened.
- Re-run with another push:
git commit --allow-empty -m rerun && git push. Or re-run a finished run withkeyward --reason "<the task>" -- gh run rerun <run-id>(gh run list --branch <branch>for the id). pushis load-bearing;workflow_dispatchalone will NOT work for a branch-only probe. GitHub only registers aworkflow_dispatchworkflow if the file is on the default branch, sogh workflow run <wf>.yml --ref <branch>errors ("no workflows found") and it never appears in the Actions UI. Keep the entry for when the probe graduates tomain; don't rely on it before then.- Do NOT open a PR just to get CI. A PR signals "ready to land," which a prototype is not.
- Omit any
pull_request:trigger — it ties runs to PR state, which is what you're avoiding.
The harness shape
Keep the probe self-contained under tests/<probe-name>/, mirroring the existing ones:
- A generator / runner (e.g. an SBPL profile generator + a
sandbox-execrunner; a.cmdresolver harness). - Fast unit + smoke tests asserting the enforcement and the bypass/fail-closed cases.
- A
README.md(what it validates, how to reproduce locally) and aresults.md(findings, plus heavy runs reproduced on demand). - The branch-scoped workflow
.github/workflows/<wf>.yml.
Mirror tests/sandbox-macos-writeconfine/ + .github/workflows/sandbox-macos-writeconfine.yml, or its Windows counterpart tests/sandbox-win-probes/ (each on its own probe branch, not main).
Keep CI lean — fast, deterministic core only: unit tests + the enforcement/bypass smoke matrix, no network, no mega-fixture. Heavy or combinatorial runs are documented in results.md and reproduced on demand. CI capacity is shared; a 22-minute-per-push probe job is already a lot.
Test every candidate FIX in one build-free run, not one per push
When the probe fails and you have several plausible repairs, do not guess-and-push: each loop costs a
full build plus queue time. Write a job that tries all candidates at once against a stand-in, with
no compile step — a .cmd/shell quoting question needs cmd.exe and a two-line stub, not the real
binary. Minutes instead of half an hour, and the losing candidates are results you keep.
Two rules that make the output trustworthy:
- Include the CURRENT broken form as a control. It must fail. If it passes, the reproduction is wrong and every other row is meaningless — say so in the output rather than reading the winners.
- A losing alternative is a result, not a job failure. Exit non-zero only when the control passes or when NO candidate works. Otherwise the job sits permanently red over a candidate you never intended to ship, and readers learn to ignore it.
Run and watch
- Kick a run by pushing to the branch; list with
gh run list --workflow <wf>.yml --branch <branch>. - Await a specific run with the
ci-watchskill (scripts/ci-watch.ts) rather than rawgh run watch— it waits for the run to exist, polls authoritative terminal status, fails fast, and exits 0 only on confirmed success. - A failure is immediately actionable — read the job log, fix the harness, push again.
Lifecycle
- The branch is the durable home of the probe while it's exploratory. Push, run, iterate.
- When it graduates into a permanent regression check, fold it into
mainthrough the normal flow (it'stests/**+ a workflow file — a content/CI change routed straight tomain). Decide its steady-state trigger then. - If you only needed the one-time answer, leave the branch as the record (or delete it once
results.mdcaptures the findings) — never open a PR to "preserve" a throwaway probe.
A cross-compile CHECK is not a test run
cargo check -p nub-cli --all-targets --target x86_64-pc-windows-gnu proves the code COMPILES for Windows. It says nothing about whether the tests PASS there, and the gap is exactly where platform behavior lives: .cmd shims vs #!/usr/bin/env node, path separators, NODE_OPTIONS tokenizing, process.title, mode bits that are no-ops.
A cross-compile is a necessary pre-flight, never the evidence. If you are un-gating tests from #[cfg(unix)], or writing anything whose behavior could differ by platform, run a branch probe BEFORE pushing — it needs no PR and costs one workflow run.
When to reach for this vs the alternatives
- Local host probe (
ad-hoc-test) — the behavior reproduces on your dev machine. Default for anything not platform-gated. - Docker (AGENTS.md) — a Linux corner: musl/glibc, a Node floor, a clean dependency-free environment, first-run install.
- This skill (CI branch probe) — a macOS or Windows behavior, or a real multi-runner matrix, that neither the host nor Docker can show.
04 同仓库更多 Skills
nub
暂无简介
nub
This project uses nub, a single Rust CLI that runs on the project's installed Node and replaces the everyday Node toolchain. ALWAYS prefer nub over node, bun, npm, npx, pnpm, and yarn for: executing JS/TS files, running package.json scripts, the npx-equivalent, installing dependencies, watch mode, and Node version management. Surfaces whenever you would otherwise reach for node/bun/npm/npx or a package manager.
worktree
Create and manage isolated git worktrees for parallel build/test/landing work on the nub repo. Invoke (via the Skill tool) whenever you need a fresh worktree to land a change, when you want to know what `.worktreeinclude` does or how to add an entry, or when cleaning up after a merge. Encodes the one-command setup (`nub scripts/new-worktree.ts <slug>` or `node …`) that bakes in the proven recipe — worktree off origin/main (vendor/aube is plain in-tree files now, no submodule init), the shared CARGO_TARGET_DIR (`~/.cache/nub/shared-target`) that all worktrees reuse, and applying `.worktreeinclude` — plus the eagerly-pull-the-shared-tree discipline and the safe cleanup path. Pairs with the `dev-loop` build skill.
windows-vm-test
Run ad-hoc Nub tests and debugging probes in the maintainer's existing local Windows 11 ARM64 QEMU/HVF VM when its external disk and runbook are present. Use for interactive win32-arm64 or Windows-on-ARM x64-emulation checks; use ci-adhoc-test for true x64-native behavior or when the local VM is unavailable.
visual-review
Verify UI/layout/styling changes are correct by computing occlusion, clipping, and alignment from the browser's resolved paint order via the chrome-devtools MCP `evaluate_script` tool — instead of eyeballing a flat screenshot. Invoke BEFORE declaring any UI, site, or styling/layout change correct. Screenshots have no depth buffer, so z-index/occlusion/clip bugs are exactly where "just look at it" fails; the `evaluate_script` routines below turn those fuzzy visual judgments into deterministic measurements — including optical center-of-mass, which measures the glyph ink's true visual center so differently-sized labels can be aligned by more than eye.
type-declarations
Add, audit, or update @nubjs/types declarations for Nub runtime APIs. Use whenever Nub adds or changes a user-visible global, built-in method, module loader type, or TC39 proposal; when TypeScript or @types/node gains an overlapping declaration; and before every release to reconcile runtime changes since the previous tag. Covers conflict-safe global augmentation, compiler-version routing, fixture matrices, and package verification.