validate-skills
Validate that commands documented in skill files actually work. Use when creating, updating, or reviewing skills to ensure all documented commands exit with code 0.
- License 许可
- MIT
- 所属仓库
- microsoft/vstest
- 最近更新
- 2026-11-14
- Stars
- ★ 977
- Forks
- 359
01 前言 Front Matter
- name
- validate-skills
- description
- Validate that commands documented in skill files actually work. Use when creating, updating, or reviewing skills to ensure all documented commands exit with code 0.
03 SKILL.md 正文
Validating Skills
Verify every executable command in a skill runs successfully on the current OS.
When to Use
- After creating or updating a skill that contains executable commands
- During skill review to catch stale or broken instructions
- When switching OS (e.g. Windows → Linux) to confirm cross-platform commands
Procedure
1. Detect Current OS
Determine which platform commands to extract:
# PowerShell (Windows)
$os = "Windows"
# Bash (Linux / macOS)
OS=$(uname -s) # "Linux" or "Darwin"
2. Extract Commands
Parse the target skill's SKILL.md and list every shell command for the detected OS:
- Many skills document commands in tables with Windows and Linux / macOS columns. Pick the column matching your OS.
- If a command contains comments like
# Windowsor# Linux / macOS, only run the one for your OS. - Placeholder substitution: Replace obvious placeholders (e.g.
<path-to-csproj>,<skill-name>) with real values from the repo. If no sensible value exists, skip the command. - Adapt cross-platform commands: Commands like
ls -lashould be adapted to PowerShell equivalents (Get-ChildItem) on Windows when no native Windows command is documented.
3. Track Results
Use the SQL tool to create a tracking table:
CREATE TABLE skill_commands (
id INTEGER PRIMARY KEY AUTOINCREMENT,
skill TEXT NOT NULL,
command TEXT NOT NULL,
expected_exit INTEGER DEFAULT 0,
actual_exit INTEGER,
status TEXT DEFAULT 'pending',
notes TEXT
);
4. Run Each Command
For every extracted command:
- Run it from the repo root
- Record the exit code
- Classify the result:
- Exit 0 → PASS
- Non-zero + environment issue (missing SDK, no internet) → ENV_ISSUE
- Non-zero + command/docs wrong → ERROR
5. Safety Rules
CRITICAL: Never run unfiltered integration/acceptance tests. They take hours.
test.sh --integrationTestortest.cmd -IntegrationMUST include a--filteror-pflag.test.sh -p smokeis acceptable (scoped to smoke tests), but expect it to be slow.
6. Report
After all commands finish, print a summary:
=== Skill Validation Report ===
Skill: <skill-name>
OS: <Linux|Darwin|Windows>
Commands tested: N
PASS: X
ENV_ISSUE: Y (list with reasons)
ERROR: Z (list failed commands with exit codes)
7. Fix or Flag
- ERROR (documentation bug): Update the skill's
SKILL.mdto fix the command. - ENV_ISSUE: Add a troubleshooting note to the skill if the environment prerequisite is not already documented.
- PASS: No action needed.
Ordering Tips
- Run restore/build before tests (tests depend on build output)
- Run the cheapest commands first to fail fast
- Batch independent test commands in parallel when possible
04 同仓库更多 Skills
vstest-build-test
Build, test, and validate changes in the vstest repository. Use when building vstest projects, running unit tests, smoke tests, or acceptance tests, or when deploying locally built vstest.console for manual testing.
trx-analysis
Parse and analyze Visual Studio TRX test result files. Use when asked about slow tests, test durations, test frequency, flaky tests, failure analysis, or test execution patterns from TRX files.
expert-reviewing
Route code and dependency changes to relevant review dimensions based on affected files and folders. Use when reviewing PRs, including Dependabot updates, GitHub Actions and generated workflow changes, or specific vstest subsystems.
creating-skills
Create custom agent capabilities when discovering novel tools, receiving task-agnostic tips from reviewers, or after researching specialized workflows not covered in existing instructions. Teaches structure, YAML optimization for LLM discoverability, and token efficiency.