AGENT SKILL

incident-response

Production incident playbook. Use immediately when production is degraded or down. Five phases: detect, contain, diagnose, resolve, post-mortem. Prevents the chaos of uncoordinated response and ensures every incident produces lasting improvements.

License 许可
Apache-2.0
最近更新
2026-11-14
Stars
★ 1.6k
Forks
491

01 前言 Front Matter

name
incident-response
description
Production incident playbook. Use immediately when production is degraded or down. Five phases: detect, contain, diagnose, resolve, post-mortem. Prevents the chaos of uncoordinated response and ensures every incident produces lasting improvements.

03 SKILL.md 正文

Incident Response

The Law

CALM IS A SKILL. PANIC IS A CHOICE.
An incident is a system behaving unexpectedly. It is solved by evidence, not speed.
Fast wrong fixes extend incidents. Slow correct fixes end them.

When to Use

Use this skill immediately when:

  • Error rate spikes above baseline
  • Latency exceeds SLA
  • A service is unreachable
  • Data is inconsistent or missing
  • A security event is suspected

The Five Phases


Phase 1: Detect & Declare (first 5 minutes)

Assess impact — answer three questions:

  1. What is broken? (which service, endpoint, feature)
  2. Who is affected? (all users, subset, specific region, specific plan tier)
  3. When did it start? (exact time from monitoring, not "someone noticed")

Declare the incident:

  • Create an incident channel: #incident-YYYY-MM-DD-short-description
  • Post the initial status: service, impact, start time, who is investigating
  • Set the incident severity:
Severity Definition
SEV-1 Complete service outage or data loss. All hands.
SEV-2 Critical feature broken, significant user impact. Team leads involved.
SEV-3 Partial degradation, workaround exists. Engineering only.
SEV-4 Minor issue, low user impact. Standard ticket.

Do not investigate alone for SEV-1/SEV-2. Assign roles:

  • Incident Commander — coordinates response, owns communication
  • Technical Lead — drives diagnosis and fix
  • Comms — updates status page and stakeholders

Phase 2: Contain (first 15 minutes)

Stop the bleeding before diagnosing the wound.

Look for the fastest way to reduce impact — in this order:

  1. Rollback — was there a recent deploy? Roll it back. Rollback is always faster than a hotfix.

    git revert HEAD --no-edit && git push
    # or
    kubectl rollout undo deployment/api
    
  2. Feature flag off — can the broken feature be disabled without affecting the whole service?

  3. Circuit breaker / rate limit — if a downstream service is causing the issue, can you stop calling it temporarily?

  4. Scale up — if it's a load issue, can you add capacity immediately?

  5. Redirect traffic — failover to a secondary region or static error page

Containment does not require knowing the root cause. Act fast to reduce impact, then diagnose.


Phase 3: Diagnose (parallel with or after containment)

Work from evidence. No guessing.

Evidence sources in order of reliability:

  1. Monitoring dashboards — what changed at the incident start time?
  2. Error logs — what errors appear that didn't before?
  3. Recent deploys — what changed in the last 24 hours?
  4. Recent config changes — feature flags, environment variables, infrastructure changes
  5. External dependencies — is a third-party service degraded? Check their status page.

Diagnostic commands:

# Recent deploys
git log --oneline -20

# Error spike in logs
grep ERROR app.log | grep "$(date +%Y-%m-%dT%H)" | wc -l

# Slow queries
psql -c "SELECT query, mean_exec_time FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"

# Service health
kubectl get pods --all-namespaces | grep -v Running
kubectl describe pod <failing-pod>
kubectl logs <failing-pod> --previous

Form a single hypothesis: "The incident is caused by X because evidence Y shows Z." Test it minimally before acting on it.


Phase 4: Resolve

Apply the fix with the minimum blast radius.

Fix principles:

  • Prefer rollback over hotfix — hotfixes under pressure introduce new bugs
  • Apply to staging first if a <5 minute delay is acceptable
  • Have a rollback plan for the fix itself
  • One change at a time — never apply multiple fixes simultaneously

After applying fix:

  • Verify the metric that triggered the incident is recovering
  • Confirm no new errors introduced
  • Keep monitoring for 15 minutes before declaring resolved

Declare resolved:

RESOLVED — [time]
Service restored. Root cause: [one sentence].
Follow-up actions tracked in post-mortem.

Phase 5: Post-Mortem (within 48 hours)

Every SEV-1 and SEV-2 gets a written post-mortem. Every SEV-3 gets at least a brief one.

Template: docs/postmortems/YYYY-MM-DD-short-title.md

# Post-Mortem: [Title]

**Date:** YYYY-MM-DD
**Severity:** SEV-N
**Duration:** Xh Ym (detected HH:MM → resolved HH:MM)
**Impact:** [who was affected and how]

## Timeline

| Time | Event |
|------|-------|
| HH:MM | First alert triggered |
| HH:MM | Engineer notified |
| HH:MM | Incident declared |
| HH:MM | Containment applied |
| HH:MM | Root cause identified |
| HH:MM | Fix deployed |
| HH:MM | Incident resolved |

## Root Cause

[One clear sentence. Not "the server crashed" — why did the server crash.]

## Contributing Factors

- [What made this possible / what made detection slow / what made recovery slow]

## What Went Well

- [Things that helped, even small ones]

## Action Items

| Action | Owner | Due |
|--------|-------|-----|
| Add alert for X | @person | YYYY-MM-DD |
| Fix root cause Y | @person | YYYY-MM-DD |
| Add runbook for Z | @person | YYYY-MM-DD |

Post-mortem rules:

  • Blameless — systems fail, not people
  • Focus on systemic improvements, not individual mistakes
  • Every action item has an owner and a date
  • Action items are tracked as real tickets, not just documented and forgotten

Communication Templates

Status page update (during incident):

[INVESTIGATING] We are investigating reports of [issue] affecting [scope].
Our team is actively working on a resolution. Next update in 30 minutes.

Resolution notice:

[RESOLVED] The issue affecting [scope] has been resolved as of [time].
Root cause was [brief explanation]. A full post-mortem will be published within 48 hours.
We apologise for the inconvenience.

04 同仓库更多 Skills

hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-review-with-annotations

Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-find-and-synthesize-context

Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-update-docs

Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-records

Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-interactive-html

Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-collaborate-threads

Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.

★ 1.6k