incident-triage-runbook
The SRE team's runbook for triaging production latency and error-rate incidents. Use this whenever investigating an incident, a latency spike, elevated error rates, or when asked "what caused X" about a production service.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-05
- Stars
- ★ 2.3k
- Forks
- 713
01 前言 Front Matter
- name
- incident-triage-runbook
- description
- The SRE team's runbook for triaging production latency and error-rate incidents. Use this whenever investigating an incident, a latency spike, elevated error rates, or when asked "what caused X" about a production service.
03 SKILL.md 正文
Incident triage
If you change the order below, say why in #sre.
Order of operations
- Pull deploys for the last 6h. Don't open the log first.
- Line the deploy timestamps up against
p99_latency_ms/error_ratefor the paged service. State the gap ("deploy 14:31, p99 moves 14:33"). - If a deploy lines up: pull the diff, read it. Check for the stuff in the next section.
- Then grep the log to confirm. Don't grep to fish.
- No deploy lines up → the service may be a victim. Check
db_pool_utilizationacross checkout/cart/auth/inventory, then the upstream deps (db-primary,db-inventory,idp.internal). Find what moved first.
Ask get_metrics for a name that doesn't exist and it lists the ones that do.
Things that have burned us
In rough order of how often:
- per-row query where there used to be a batch
- cache decorator removed "temporarily"
- new query, no index
- blocking call in an async handler
- retry loop with no backoff
Write-up
One line at the bottom of your reply:
Root cause:
<sha>— one sentence on the mechanism.
If it wasn't a deploy, put the component or upstream dep where the sha goes (db-primary, stripe-api, whatever). Still one sentence.
Everything above that line is evidence. Keep it short; the long version goes in the postmortem doc.
If you were handed an incident id, also save the structured version to /workspace/findings/<INCIDENT_ID>.json so the postmortem can be built from it:
{
"incident_id": "INC-0000",
"service": "checkout",
"title": "one line, what users saw",
"root_cause": {"ref": "<sha or component>", "summary": "one sentence on the mechanism"},
"timeline": [{"time": "14:31:18", "event": "what happened"}],
"evidence": ["one fact per line, with the number or timestamp that proves it"],
"ruled_out": ["what you checked that turned out not to be the cause, and why"],
"impact": {"metric": "p99_latency_ms", "baseline": 65, "peak": 3640, "unit": "ms"},
"fix": "the immediate remediation",
"caused_by_incident": null
}
Set caused_by_incident to another page's id when your service is a victim of it.
04 同仓库更多 Skills
postmortem-report
How the SRE team writes a postmortem PDF. Use this whenever asked for a postmortem, an incident report, a write-up to share, or a PDF of what happened.
eval-audit-and-sweep
暂无简介
edgartools-sec-data
How to pull SEC EDGAR data with the edgartools Python package — company lookup, filings, XBRL financial statements, and filing sections like Item 1A risk factors. Use whenever a task involves SEC filings, 10-K/10-Q data, or company financials.
workshop
Workshop coach for the Research Desk (SEC agents) workshop. Use when the user types /workshop, asks for a workshop act or module ("act 2", "next act", "where am I"), wants a TODO(workshop-N) implemented or explained, or asks for help following WORKSHOP.md.
weekly-report
Structure and data sources for the weekly inventory report. Load this when the task is "weekly report", "Monday report", or "summarize inventory status".
supplier-selection
How to rank and pick a supplier for a SKU. Load this whenever a task involves choosing a supplier, comparing quotes, or creating a purchase order.