AGENT SKILL

catalog-probe

暂无简介

License 许可
MIT
所属仓库
nubjs/nub
最近更新
2026-11-14
Stars
★ 4.4k
Forks
71

03 SKILL.md 正文

Running the build-jail catalog probe

The probe measures the minimum capability grant a package's lifecycle scripts need, by walking a 54-state capability space in ascending cost order and taking the first state that reproduces an unjailed control. Its output is the build-jail catalog.

Everything here is a failure that has already happened. None of it is hypothetical.

Before you run anything

1. Build with the override feature, or nothing works.

scripts/rust-build.sh build -p nub-cli --profile fast \
  --features nub-cli/build-jail-catalog-override

Any cargo command on a profile rewrites that profile's binary with ITS features. A cargo test --profile fast in another shell silently strips the override and every subsequent package records a control failure. run-batch.sh snapshots the binary to defend against this, but a bare nub invocation outside the batch still uses the live one.

2. Never edit the harness while a batch is running. Each package is a fresh node search.mjs invocation, so an edit mid-sweep changes the harness under the remaining packages. This has produced a sweep where the first half and the second half were measured by different code — and, in the worst case, 54 of 100 packages crashed because the file changed beneath them.

3. Run one cheap package first as an instrument check.

./run-batch.sh <nub> --force is-odd@3.0.1     # expect verdict=MINIMUM, state=(nothing), 2 cells

If that is not clean, nothing after it means anything.

Running a sweep

./run-batch.sh <nub> --file worklist.txt          # a worklist, one pkg@version per line
./run-batch.sh <nub> --force <pkg>@<version>      # one package, --force re-measures

Long sweeps go in a background shell (run_in_background: true), never a foreground call and never nohup/setsid — a detached run cannot be tracked and never wakes you.

Reading the results — coverage first, always

node watch-sweep.mjs results/runs <since-ms> worklist.txt

Pass the worklist. Without it you get a summary of what succeeded and no idea what did not run. The single most expensive mistake made with this tool was reporting the survivors of a sweep as its result: 54 of 100 packages produced no record, the batch discarded their stderr, and the remaining 46 looked like a finished corpus. The failures are not randomly distributed — heavy native builds fail most, and those are exactly the packages most likely to need a grant, so the surviving sample is biased toward "needs nothing."

Rules for reading a sweep:

  • attempted / recorded / FAILED is the headline, printed at the end of every batch. If FAILED is not zero, you do not have a corpus.
  • A run of identical failures indicts the HARNESS, not the packages. Check the FIRST one and fix that; the other ninety-nine are the same fault repeated.
  • Read the first error in a log, never the last. A node stack trace ends with the version banner, and an install log ends with a summary — the cause is usually ~40 lines earlier. Five successive wrong diagnoses of one package all came from reading the tail.

The instrument has no test — the failure with no symptom

A change to the fixture or to baseline.json is a change to the measuring instrument. Its failure mode is not an error; it is that every package measures as needing NOTHING. Every verdict MINIMUM, coverage 100%, nothing fails. Three times in one session:

  • A hand-written package-lock.json with an empty packages map — nub believed the project had no dependencies. Puppeteer's control fell from 9,629 installed files to 32.
  • A baseline entry using $home/... — the wrong grammar (see sentinels below), so the jail failed to compile and no lifecycle script spawned. Surfaced as failed to spawn, which reads as a nub defect.
  • Worst: a measurement taken during the second window was written up as a finding — "this package's grant dissolved" — when the jail was simply not running.

All three were caught by disbelieving the number, never by a check. Eight packages needing nothing, including ones that cannot work without downloading a binary, is not a measurement.

The pre-flight now runs a FIXTURE CANARY: puppeteer@25.4.0 must install >5000 files and be materialized, or the batch refuses. It asserts the control's SHAPE, not a verdict — a package that legitimately needs nothing looks identical either way, so is-odd cannot catch this. NUB_PROBE_SKIP_CANARY=1 disables it when deliberately testing the fixture.

Never report a measurement taken while the harness was known-broken. Re-run it first.

Three $ vocabularies, and they are not interchangeable

Namespace Valid names Used by
Compiler fs sentinels $cache, $tmp (closed set), plus ~/ baseline paths, catalog fs rules
Harness path tokens $proj/, $store/, $home/ recorded paths, writePaths entries
Network host sets $<name> on the net axis net rules only

$home is meaningful in the second and invalid in the first. The compiler rejects an unknown sentinel by name and lists the valid ones — that message is what makes this a one-step diagnosis.

Verdicts

Verdict Means Do
MINIMUM Measured. state is the minimal grant. Nothing.
HARNESS-CRASH / HARNESS-TIMEOUT The probe itself failed. Read harness-stderr.log beside the record. Never a package fact.
HARNESS-ERROR The catalog override did not engage in the control. Wrong binary, or the harness emits a catalog shape the parser rejects.
BROKEN-IN-ENVIRONMENT Fails under npm too, same signature. Grant nothing. Check needsInvestigation.
BROKEN-EVEN-WITH-EVERYTHING Fails jailed at the widest grant, but npm succeeds. A nub defect — the most valuable output. Never a grant gap.

Changing the catalog shape — five places move together

Written in one place, read in four. Missing one fails as something else entirely: a shape change that reached the parser but not the harness produced a hundred-package sweep in which every package reported that the override had not engaged, which reads as a broken binary.

  1. crates/nub-sandbox/src/catalog_v2.rs — types, parse, validation, resolution
  2. crates/nub-sandbox/src/catalog_override.rs — grant count and lookup
  3. tests/build-jail-search/collate.mjs — writes the catalog
  4. catalogFor in tests/build-jail-search/search.mjs — synthesizes a catalog per cell, every run
  5. tests/build-jail-search/overrides/ — hand-written entries

Plus the --selftest assertions, which read the synthesized shape and will silently pass on the wrong one if not updated.

The pre-flight probe catalog must come from catalogFor, never a literal. It is emitted by search.mjs --emit-sample-catalog for exactly this reason. A hand-written probe drifts from what the harness emits, and a catalog with an empty package map is the worst possible probe because it parses under every shape there has ever been.

The oracle, and why it is shaped this way

  • Judge the ARTIFACT, not the exit code. A cell passes only if it reproduces the control on exit code and on the digest of the sorted path list. A hook installer that cannot see the project writes zero of seventeen hooks and exits 0.
  • The control runs TWICE, combined by UNION. Never intersection — that compares on fewer paths, so a cell that failed to write an unstable path still passes and the recorded minimum is too narrow, which is the exact failure the jail exists to avoid.
  • Every other package is held at full grant, so the package under test is the only variable.
  • When the oracle says something impossible, suspect the oracle. "Failed all 55 cells, nondeterministic" was 3 varying paths out of 2,734 — all one timestamped log filename.

Ground truth

  • The tarball manifest, not the packument. They disagree: fsevents@2.3.3's packument declares install: node-gyp rebuild and its published tarball does not. nub runs the tarball.
  • Prefer a global baseline/env entry over a per-package grant or a harness filter. A filter hides one tool's write after the fact and must be re-derived per tool. Two entries already earn their place: PYTHONDONTWRITEBYTECODE=1 and npm_config_logs_max=0, each of which stops a write happening at all rather than filtering it afterwards.
  • Over-granting is the safe direction. The failure to avoid is packages breaking.

Related

  • wiki/design/build-jail.md — the canonical design: capability model, bands, placement
  • .frizz/build-jail-catalog-schema.md — the catalog schema spec
  • rust-build — cargo mechanics and the profile/feature trap

04 同仓库更多 Skills

nubjs
nubjs/nub

nub

暂无简介

★ 4.4k
nubjs
nubjs/nub

nub

This project uses nub, a single Rust CLI that runs on the project's installed Node and replaces the everyday Node toolchain. ALWAYS prefer nub over node, bun, npm, npx, pnpm, and yarn for: executing JS/TS files, running package.json scripts, the npx-equivalent, installing dependencies, watch mode, and Node version management. Surfaces whenever you would otherwise reach for node/bun/npm/npx or a package manager.

★ 4.4k
nubjs
nubjs/nub

worktree

Create and manage isolated git worktrees for parallel build/test/landing work on the nub repo. Invoke (via the Skill tool) whenever you need a fresh worktree to land a change, when you want to know what `.worktreeinclude` does or how to add an entry, or when cleaning up after a merge. Encodes the one-command setup (`nub scripts/new-worktree.ts <slug>` or `node …`) that bakes in the proven recipe — worktree off origin/main (vendor/aube is plain in-tree files now, no submodule init), the shared CARGO_TARGET_DIR (`~/.cache/nub/shared-target`) that all worktrees reuse, and applying `.worktreeinclude` — plus the eagerly-pull-the-shared-tree discipline and the safe cleanup path. Pairs with the `dev-loop` build skill.

★ 4.4k
nubjs
nubjs/nub

windows-vm-test

Run ad-hoc Nub tests and debugging probes in the maintainer's existing local Windows 11 ARM64 QEMU/HVF VM when its external disk and runbook are present. Use for interactive win32-arm64 or Windows-on-ARM x64-emulation checks; use ci-adhoc-test for true x64-native behavior or when the local VM is unavailable.

★ 4.4k
nubjs
nubjs/nub

visual-review

Verify UI/layout/styling changes are correct by computing occlusion, clipping, and alignment from the browser's resolved paint order via the chrome-devtools MCP `evaluate_script` tool — instead of eyeballing a flat screenshot. Invoke BEFORE declaring any UI, site, or styling/layout change correct. Screenshots have no depth buffer, so z-index/occlusion/clip bugs are exactly where "just look at it" fails; the `evaluate_script` routines below turn those fuzzy visual judgments into deterministic measurements — including optical center-of-mass, which measures the glyph ink's true visual center so differently-sized labels can be aligned by more than eye.

★ 4.4k
nubjs
nubjs/nub

type-declarations

Add, audit, or update @nubjs/types declarations for Nub runtime APIs. Use whenever Nub adds or changes a user-visible global, built-in method, module loader type, or TC39 proposal; when TypeScript or @types/node gains an overlapping declaration; and before every release to reconcile runtime changes since the previous tag. Covers conflict-safe global augmentation, compiler-version routing, fixture matrices, and package verification.

★ 4.4k