openclaw-testing

OpenClaw Testing

Proof policy

• PR CI = default broad proof. Change builds + focused tests pass → open ready PR. CI runs broad suites; never delay PR for pre-PR suite runs.

• Build unavailable before publication: when publication is already authorized, remote capacity is unavailable, and local execution is unqualified, push/open a draft PR with the exact missing build and capacity gap recorded. Defer that build to exact-head CI. Require its success before marking ready or merging. This changes proof timing only, not publication authority, source trust, credentials, owner approval, or merge gates. Automatic CI skips drafts. For a trusted canonical branch, use the existing ci.yml manual dispatch with target_ref set to the full pushed head SHA (ghx workflow run ci.yml --repo openclaw/openclaw --ref main -f target_ref=<sha>). Verify the selected checkout SHA and successful build-artifacts job cover the required build. CI's build:ci-artifacts omits full declaration emission. Keep uncovered proof pending. A skipped job or workflow-ref SHA is not build proof. Link the run in the PR, then use normal ready-PR CI and landing gates. Untrusted source keeps its secretless route.

• Before PR, prove narrowly: touched tests with pnpm test <file> --maxWorkers=1, targeted typecheck, lint, format. Prefer local proof for trusted, bounded work when dependencies and current host capacity are qualified. Local execution is optional: a focused command can still load a large compiler graph or expand to broad lanes.

• Choose remote proof deliberately. Use PR CI for broad proof; Crabbox/Testbox may cover expensive checks, a busy or unqualified local host, another OS/device, live providers/channels, long E2E, or heavy benchmarks. Acquire only when the command is ready and capacity is granted promptly. If queued, stop only the task-owned allocation; use PR CI or a safe local scope. Keep uncovered platform-specific proof pending. Never force heavy work onto a busy machine or chain providers. Use the smallest profile proven for the selected workload.

• Never rerun or re-push just for green. Fix real failures; read available CI failure classification before attributing red.

• Prove changed contract, complete required gates, finish. Broaden/repeat only for changed inputs, failures, or named unresolved risk.

• Follow scoped AGENTS.md; authoring/reviewing tests: use boundary guide and test-audit.

Select the proof

• Select proof for touched contract; reuse valid proof (rerun for changed inputs or missing coverage); finish workflow's required gates within user/host limits; report unrun checks and gaps. Prove user-visible behavior through real flow when feasible; external APIs need live contract proof; isolated mock-Gateway harness = valid channel boundary proof. Docs-only: docs sanity and git diff --check.

• Runtime defect: reproduce real entry point; rerun it + relevant siblings.

• Trusted source: pnpm changed:lanes --json, focused tests + targeted typecheck/lint/format; pnpm check:changed = optional smart local check.

• Public SDK/plugin contract: representative consumers, not all-plugin sweep.

• Build output, lazy imports, package boundaries: include pnpm build or the repository CI build that covers the changed contract. Record uncovered packaging/platform/declaration requirements. Required full package or release builds still need their matching proof.

• Workflow: git diff --check + pnpm check:workflows.

• Docs only: relevant docs/link/format checks + git diff --check; no runtime tests.

• Read on demand: local commands, Crabbox, OpenClaw remote setup, package/Docker proof, release CI, plugin release matrix, Docker authoring, channel skill/ Control UI E2E.

• An isolated mock-Gateway harness is valid channel boundary proof; state live gaps. Prove user-visible behavior through the real flow when feasible; external APIs require live contract proof. Read the scoped channel/platform guide for live-channel or native-app work. Telegram claims require Test Server userbot proof with Convex-leased credentials. Platform claims require real-device evidence; macOS permission proof requires a stable signed app.

• Release proof: preserve candidate/Tooling SHAs; never grants publication authority.

Source and state boundaries

• Never execute untrusted contributor/fork code, wrappers, config locally. Use secretless fork CI or sanitized direct AWS through Crabbox; never credential-hydrated Testbox. Credentialed execution: maintainer approval after review required; never hydrate untrusted lease.
• Isolated state + free port. Never restart, edit, or test operator Gateway/real data without explicit per-task approval. Preserve unrelated processes and shared dependency installs.
• Concurrent test/check commands: separate OPENCLAW_VITEST_FS_MODULE_CACHE_PATH values or run serially; checks can run Vitest too. Repository wrappers only, never raw Vitest. Prepared linked worktrees: node scripts/run-vitest.mjs <file> --maxWorkers=1 avoids pnpm reconciliation; PR CI runs the broad checks.

CI failures

• PR CI fails → pnpm test:ci-failed <pr>; read pre-existing/new/unknown classification.
• Fix new failures; investigate unknown → pnpm test:ci-failed <pr> --run until selected tests pass locally → push once.
• --run enables OPENCLAW_NODE_TEST_PLAN_CONTINUE_ON_FAILURE=1: collect ordinary failures across selected configs, while infrastructure failures still stop; use the same opt-in for manual focused reruns.
• Pre-existing failures: failure-table landing rule. Matching signatures ≠ job-name-only attribution.
• Never gh run rerun to chase green; never local full suite for CI repair.
• Bind diagnosis to exact SHA + job. Fetch failed logs once; prefer exact run/job state over stale PR rollups.
• Distinguish product, harness, infrastructure, credentials, superseded-run cancellation. Snapshots pass macOS/fail CI: reproduce Linux/Node bytes before regenerating.

Baseline and expected-failure exceptions

Narrow baseline, snapshot, ignore, and expected-failure exceptions are implementer decisions only when each entry is verified against source and the PR records its reason (for example, tool-owned metadata in an upgrade snapshot while user data stays exact). Never mask a real regression or user-data change. Blanket or unverified exceptions need approval. Exact shrink-only ratchet updates are maintenance. A fix PR may add a separate tracked config/test-lines-allocations/<change>.txt record for exactly its own justified net regression-test lines when the squash body states the reason. Keep the shared scalar unchanged so independent grants merge. The checker bounds effective ceiling growth against actual LF growth and preserves inherited debt. Other raises, including scalar-only reconciliation, need approval. See the cost budget for record format and pruning.

For module mocks, follow the fixture contract. When an isolation annotation retires an existing baseline entry, commit the annotation and corresponding baseline removal together. Inspect the full baseline diff because --prune scans the whole checkout. Verify the updated baseline with a guard run without --prune.

Test failure policy

• Failures = defects. Bounded reproduction attempt (same shard order first); identify owner; fix product, fixture, shared-state, ordering bugs.
• Proven repair: add regression; cite another owner's fix when applicable.
• Safe fix cannot be established/completed: record original failure, attempts, evidence, uncertainty in PR; continue under normal CI/review gates.
• Unresolved failure alone neither blocks landing nor requires extra approval. Passing replay ≠ proven fix.
• Never hide failures with retries, longer timeouts, weaker assertions, broader mocks, altered baselines.