AGENT SKILL

detect-flaky

Detect, diagnose, and fix flaky tests in Python, Go, or JavaScript/TypeScript codebases by repeated execution + root-cause analysis. Triggers on: "detect flaky tests", "find flaky tests", "diagnose flaky tests".

License 许可
MIT
最近更新
2026-10-26
Stars
★ 0
Forks
0

01 前言 Front Matter

name
detect-flaky
description
Detect, diagnose, and fix flaky tests in Python, Go, or JavaScript/TypeScript codebases by repeated execution + root-cause analysis. Triggers on: "detect flaky tests", "find flaky tests", "diagnose flaky tests".
argument-hint
[test-command]

03 SKILL.md 正文

/detect-flaky — Flaky Test Detection and Diagnosis Agent

This agent detects, diagnoses, and fixes flaky tests in Python, Go, and JavaScript/TypeScript codebases.

A test is flaky when it non-deterministically passes or fails without code changes.


Mission (North Star)

Identify flaky tests through:

  1. Static pattern scanning — Regex-based detection of known anti-patterns
  2. Empirical detection — Run tests N times to catch intermittent failures
  3. Root cause diagnosis — Map findings to a curated catalog of causes
  4. Fix generation — Produce concrete before/after code fixes

Your primary outcome is deterministic, trustworthy test suites.


Invocation Examples

/detect-flaky scan this codebase for flaky test patterns

/detect-flaky focus on the Python tests, especially around the API client module

/detect-flaky run Go tests 15 times with race detector and find intermittent failures

/detect-flaky I have a test_payment_flow that fails ~10% of the time in CI, diagnose it

/detect-flaky scan elt/tests/ for timing-related issues

Pattern Catalog (Reference During Analysis)

High Severity (Frequently Flaky)

ID Language Anti-Pattern Detection Fix
CROSS-TIME All Time-dependent assertions time.Sleep, setTimeout, datetime.now() in assertions Use deterministic time injection or freeze time
CROSS-RACE All Race conditions in concurrent test code Shared mutable state across goroutines/threads Use proper synchronization or test isolation
CROSS-EXTDEP All Unmocked external dependencies requests.get(), http.Get(), fetch() in test files Mock all network calls
PY-MOCK-LEAK Python Mock not cleaned up patch().start() without stop() Use context manager or addCleanup()
PY-REAL-HTTP Python Real HTTP in tests requests, httpx, urllib calls Use responses, httpretty, or respx
PY-RANDOM-SEED Python Unseeded random random.choice(), random.randint() Set random.seed() in fixture
GO-PARALLEL-STATE Go Shared state in parallel tests t.Parallel() with package-level vars Use test-local state or sync primitives
GO-GOROUTINE-LEAK Go Goroutines outliving test go func() without cleanup Use t.Cleanup() or errgroup
GO-SETENV-PARALLEL Go os.Setenv in parallel tests Environment mutation + t.Parallel() Use t.Setenv() (Go 1.17+)
GO-RACE-CONDITION Go Data race in test Concurrent map/slice access Run with -race flag
JS-WAITFOR-TIMING JS/TS Wrong async query getBy* on async content Use findBy* or waitFor()
JS-TIMER-LEAK JS/TS Fake timers not restored jest.useFakeTimers() without restore Call jest.useRealTimers() in afterEach
JS-MOCK-RESTORE JS/TS Mock not restored jest.spyOn() without cleanup Use restoreAllMocks() in afterEach
JS-ACT-MISSING JS/TS State update outside act() Component updates without act() wrapper Wrap state-triggering code in act()

Medium Severity (Sometimes Flaky)

ID Language Anti-Pattern Detection Fix
CROSS-RANDOM All Unseeded RNG rand, random, Math.random() Seed RNG or inject deterministic values
CROSS-FLOAT All Exact float equality == on float results Use pytest.approx(), tolerance comparisons
CROSS-ENV All Environment variable mutation os.Setenv, process.env.X = Restore original values in cleanup
CROSS-ORDER All Test order dependency Tests pass alone, fail together Isolate state, use fresh fixtures
PY-FIXTURE-SCOPE Python Session/module fixtures with mutable state scope="session" with list/dict Use function scope or immutable fixtures
PY-ASYNC-LOOP Python Manual event loop management asyncio.get_event_loop(), loop.run_until_complete() Use pytest-asyncio with @pytest.mark.asyncio
PY-TEMPFILE Python Hardcoded temp paths /tmp/test_file.txt Use tmp_path fixture or tempfile
GO-CHANNEL-TIMING Go Short channel timeouts select with time.After(10ms) Use longer timeouts or remove timing dependency
GO-MAP-ORDER Go Map iteration order assertions Loop over map, assert order Sort keys first or use deterministic structure
GO-HTTP-SERVER Go Test server port conflicts Hardcoded ports in tests Use httptest.NewServer() with dynamic ports
JS-SNAPSHOT-FRAGILE JS/TS Brittle snapshots Large objects, timestamps in snapshots Exclude volatile fields, use smaller snapshots
JS-ASYNC-UNMOUNT JS/TS State update after unmount Async callback on unmounted component Use cleanup functions, abort controllers
JS-FETCH-MOCK JS/TS Incomplete fetch mock Missing error/timeout cases Mock all fetch scenarios

Low Severity (Occasionally Flaky)

ID Language Anti-Pattern Detection Fix
CROSS-HARDCODED-PATH All Hardcoded absolute paths /home/, /Users/, C:\ in tests Use relative paths or fixtures
PY-PARAMETRIZE-ID Python Non-deterministic test IDs @pytest.mark.parametrize with dicts Use explicit ids= parameter
PY-MONKEYPATCH-SCOPE Python Monkeypatch scope mismatch Monkeypatch in wrong scope Match monkeypatch scope to fixture scope
GO-TESTMAIN-EXIT Go Missing os.Exit in TestMain TestMain without os.Exit(m.Run()) Always call os.Exit(m.Run())
GO-BUILD-TAGS Go Build tag inconsistency Tests pass locally, fail in CI Ensure consistent build tags
JS-SNAPSHOT-LARGE JS/TS Large snapshot files toMatchSnapshot() on complex output Use toMatchInlineSnapshot() or targeted assertions
JS-CONSOLE-MOCK JS/TS Console methods not mocked Tests output noise, may have timing Mock console.* methods

Analysis Protocol

Phase 1: Understand the Request

Parse the user's request to determine:

  1. Scope: Specific directory, file, test name, or entire codebase?
  2. Language: Python, Go, JS/TS, or all?
  3. Mode: Static scan, empirical detection, or diagnosis of known flaky test?
  4. Symptoms: If diagnosing, what failure rate, error messages, CI context?

Phase 2: Static Pattern Scan

For static analysis, scan test files for patterns from the catalog.

Python test files: **/test_*.py, **/*_test.py, **/tests/**/*.py Go test files: **/*_test.go JS/TS test files: **/*.test.{js,ts,jsx,tsx}, **/*.spec.{js,ts,jsx,tsx}, **/__tests__/**/*.{js,ts,jsx,tsx}

For each match:

  1. Identify the pattern ID
  2. Extract the offending code snippet
  3. Explain why it causes flakiness
  4. Provide a concrete fix

Phase 3: Empirical Detection (If Requested)

Run the test suite multiple times to detect intermittent failures:

# Python
for i in {1..10}; do pytest <path> --tb=no -q 2>&1; done | tee results.txt

# Go
for i in {1..10}; do go test -race -count=1 <package> 2>&1; done | tee results.txt

# JavaScript (Jest)
for i in {1..10}; do npm test -- --testPathPattern=<pattern> 2>&1; done | tee results.txt

Analyze results for:

  • Tests that flip between pass/fail
  • Tests that timeout inconsistently
  • Tests with different failure messages across runs

Phase 4: Root Cause Diagnosis

For each identified flaky test:

  1. Map to pattern catalog entry
  2. Explain the non-determinism source
  3. Identify the specific line(s) causing flakiness
  4. Consider test dependencies and execution order

Phase 5: Fix Generation

Produce concrete fixes:

### Fix for `test_user_creation` (PY-MOCK-LEAK)

**Before:**

```python
def test_user_creation():
    mock_db = patch('app.db.connection').start()
    # test code...
    # Missing: mock_db.stop()
```

After:

def test_user_creation():
    with patch('app.db.connection') as mock_db:
        # test code...
        # Automatically cleaned up

Why this fixes it: The context manager ensures the mock is always stopped, even if the test fails or raises an exception.


---

## Tool Usage

### Scripts (If Available)

```bash
# Fast static scan
python scripts/scan_patterns.py --lang python --format text

# Empirical detection
python scripts/detect_flaky.py --lang python --runs 10 --output report.json

Direct Commands

# Python - find time-dependent code in tests
grep -rn "time.sleep\|datetime.now\|time.time" --include="test_*.py"

# Go - find parallel tests with shared state
grep -rn "t.Parallel()" --include="*_test.go" -A5 | grep -E "var |= "

# JS - find missing async handling
grep -rn "getByText\|getByRole" --include="*.test.tsx" | grep -v "await\|findBy"

# Run with race detector (Go)
go test -race -count=5 ./...

# Run pytest multiple times
pytest --count=10 -x  # requires pytest-repeat

MCP Tool Usage

Use ToolSearch with direct selection:

select:mcp__analyzer__ruff-check      # Python linting
select:mcp__filesystem__search_files  # Find test files

Output Format

Summary Report

# Flaky Test Analysis: [scope]

**Scan Date:** YYYY-MM-DD
**Files Scanned:** X test files
**Patterns Checked:** 30+
**Method:** [Static scan / Empirical (N runs) / Diagnosis]

## Findings Summary

| Severity | Count | Top Patterns                |
| -------- | ----- | --------------------------- |
| High     | X     | CROSS-TIME, PY-MOCK-LEAK    |
| Medium   | X     | CROSS-RANDOM, JS-TIMER-LEAK |
| Low      | X     | CROSS-HARDCODED-PATH        |

## High Severity Issues

### 1. [Pattern ID]: [File:Line]

**Code:**

```language
// problematic code
```

Problem: [Explanation of why this causes flakiness]

Fix:

// fixed code

Verification: [How to confirm the fix works]


Recommendations

  1. [Prioritized action item]
  2. [Prioritized action item]
  3. [Prioritized action item]

---

## CI Integration Guidance

When asked about CI integration, recommend:

```yaml
# GitHub Actions example
name: Flaky Test Check
on: [pull_request]

jobs:
  flaky-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Scan for flaky patterns
        run: |
          python scripts/scan_patterns.py \
            --min-severity MEDIUM \
            --format json \
            --output flaky-report.json
        continue-on-error: true

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: flaky-test-report
          path: flaky-report.json

Quality Standards

Every finding must be:

  • Actionable: Specific file, line, and fix
  • Justified: Clear explanation of the flakiness mechanism
  • Prioritized: Severity based on impact and frequency
  • Verifiable: How to confirm the fix works

Never report:

  • Style issues unrelated to flakiness
  • Theoretical issues without evidence
  • Issues in non-test code (unless directly causing test flakiness)

Ready State

Await user command with scope and intent:

/detect-flaky [scope] [optional: language, mode, specific test name]

04 同仓库更多 Skills

kaiohenricunha
kaiohenricunha/dotbabel

yolo

Start temporary autonomy for the current task or an explicit scope with a required --for expiry. Codex permits scoped pushes, merges, deploys, and releases after strict testing and validation; Full access is a separate runtime setting. Claude Code uses a session permission hook with a protected-action denylist.

开发工程师
★ 0
kaiohenricunha
kaiohenricunha/dotbabel

veracity-audit

Audit a data pipeline for Veracity and Value. Dispatches data-scientist, compliance-auditor, and data-engineer agents with project context injected at dispatch time. All source paths are supplied via flags — no defaults, no project assumptions. Subcommands: audit (full pipeline walk), score-check (scoring math only), gate-check (gate coverage only), source-trace <label> (single source end-to-end). Invoke when: "audit pipeline", "veracity check", "scoring math correct?", "check gates", "trace source", "verify quality gates", "formula correct?".

律师与法务Skill与智能体开发
★ 0
kaiohenricunha
kaiohenricunha/dotbabel

validate-spec

Audit an already-implemented spec against the codebase. Walks each constraint (ARCH-N, PERF-N, KD-N, etc.) and acceptance criterion, grounds findings in file:line evidence, runs the spec.json acceptance_commands, and writes a single audit doc to docs/audits/. Use whenever the user asks to "validate a spec", "audit a spec", "check if spec is implemented", "verify the spec is done", "is this spec really done", or otherwise wants closure on spec-driven work. Read-only against the spec — produces an audit, never modifies the spec itself.

★ 0
kaiohenricunha
kaiohenricunha/dotbabel

terragrunt-specialist

Deep-dive Terragrunt hierarchy review, DRY pattern audits, and run-all orchestration analysis. Use for structured investigations of multi-environment Terragrunt layouts, dependency graphs, remote state config, and hook correctness. Triggers on: "Terragrunt audit", "run-all review", "dependency block", "DRY pattern review", "env hierarchy audit", "mock_outputs", "terragrunt hooks".

运维与部署
★ 0
kaiohenricunha
kaiohenricunha/dotbabel

terraform-specialist

Deep-dive Terraform architecture review, module design, state management, and migration. Use for structured investigations of Terraform workspaces, provider configuration, module coupling, import workflows, and test coverage. Triggers on: "Terraform audit", "module review", "state management", "Terraform import", "workspace design", "provider config review", "Terraform testing".

测试与质量运维与部署
★ 0
kaiohenricunha
kaiohenricunha/dotbabel

spec

Create structured engineering specs through interactive pairing. Use when the user wants to create a spec, design doc, technical specification, architecture document, RFC, or engineering plan. Triggers on "let's spec this out", "create a spec", "design doc", "write a technical plan", "plan the architecture". Works for greenfield and brownfield projects. Outputs to docs/specs/.

设计师视觉设计
★ 0