observability-design
Monitoring strategies, distributed tracing, SLI/SLO design, and alerting patterns. Use when designing monitoring infrastructure, defining service level objectives, implementing distributed tracing, creating alert rules, building dashboards, or establishing incident response procedures. Covers the three pillars of observability and production readiness.
- License 许可
- MIT
- 最近更新
- 2026-11-01
- Stars
- ★ 3
- Forks
- 1
01 前言 Front Matter
- name
- observability-design
- description
- Monitoring strategies, distributed tracing, SLI/SLO design, and alerting patterns. Use when designing monitoring infrastructure, defining service level objectives, implementing distributed tracing, creating alert rules, building dashboards, or establishing incident response procedures. Covers the three pillars of observability and production readiness.
02 资源文件 1
- 参考文档references/monitoring-patterns.md10.3 KB
03 SKILL.md 正文
Observability Patterns
When to Use
- Designing monitoring infrastructure for new services
- Defining SLIs, SLOs, and error budgets for reliability
- Implementing distributed tracing across microservices
- Creating alert rules that minimize noise and maximize signal
- Building dashboards for operations and business stakeholders
- Establishing incident response and postmortem processes
- Diagnosing production issues through telemetry correlation
Philosophy
You cannot fix what you cannot see. Observability is not about collecting data - it is about answering questions you have not thought to ask yet. Good observability turns every incident into a learning opportunity and every metric into actionable insight.
The Three Pillars
Metrics
Numeric measurements aggregated over time. Best for understanding system behavior at scale.
Characteristics:
- Highly efficient storage (aggregated values)
- Support mathematical operations (rates, percentiles)
- Enable alerting on thresholds
- Limited cardinality (avoid high-cardinality labels)
Types:
| Type | Use Case | Example |
|---|---|---|
| Counter | Cumulative values that only increase | Total requests, errors, bytes sent |
| Gauge | Values that go up and down | Current memory, active connections |
| Histogram | Distribution of values in buckets | Request latency, payload sizes |
| Summary | Similar to histogram, calculated client-side | Pre-computed percentiles |
Logs
Immutable records of discrete events. Best for understanding specific occurrences.
Characteristics:
- Rich context and arbitrary data
- Expensive to store and query at scale
- Essential for debugging specific issues
- Should be structured (JSON) for parseability
Structure:
Required fields:
- timestamp: ISO 8601 format with timezone
- level: ERROR, WARN, INFO, DEBUG
- message: Human-readable description
- service: Service identifier
- trace_id: Correlation identifier
Context fields:
- user_id: Sanitized user identifier
- request_id: Request correlation
- duration_ms: Operation timing
- error_type: Classification for errors
Traces
Records of request flow across distributed systems. Best for understanding causality and latency.
Characteristics:
- Show request path through services
- Identify latency bottlenecks
- Reveal dependencies and failure points
- Higher overhead than metrics
Components:
- Trace: Complete request journey
- Span: Single operation within a trace
- Context: Metadata propagated across services
SLI/SLO/SLA Framework
Service Level Indicators (SLIs)
Quantitative measures of service behavior from the user perspective.
Common SLI categories:
| Category | Measures | Example SLI |
|---|---|---|
| Availability | Service is responding | % of successful requests |
| Latency | Response speed | % of requests < 200ms |
| Throughput | Capacity | Requests processed per second |
| Error Rate | Correctness | % of requests without errors |
| Freshness | Data currency | % of data < 1 minute old |
SLI specification:
SLI: Request Latency
Definition: Time from request received to response sent
Measurement: Server-side histogram at p50, p95, p99
Exclusions: Health checks, internal tooling
Data source: Application metrics
Service Level Objectives (SLOs)
Target reliability levels for SLIs over a time window.
SLO formula:
SLO = (Good events / Total events) >= Target over Window
Example:
99.9% of requests complete successfully in < 200ms
measured over a 30-day rolling window
Setting SLO targets:
- Start with current baseline performance
- Consider user expectations and business impact
- Balance reliability investment against feature velocity
- Document error budget policy
Error Budgets
The allowed amount of unreliability within an SLO.
Calculation:
Error Budget = 1 - SLO Target
99.9% SLO = 0.1% error budget
= 43.2 minutes downtime per 30 days
= 8.64 seconds per day
Error budget policies:
- Budget remaining: Continue feature development
- Budget depleted: Focus on reliability work
- Budget burning fast: Freeze deploys, investigate
Alerting Strategies
Symptom-Based Alerts
Alert on user-visible symptoms, not internal causes.
Good alerts:
- Error rate exceeds threshold (users experiencing failures)
- Latency SLO at risk (users experiencing slowness)
- Queue depth growing (backlog affecting users)
Poor alerts:
- CPU at 80% (may not affect users)
- Pod restarted (self-healing, may not affect users)
- Disk at 70% (not yet impacting service)
Multi-Window, Multi-Burn-Rate Alerts
Detect fast burns quickly, slow burns before budget depletion.
Configuration:
Fast burn: 14.4x burn rate over 1 hour
- Fires in 1 hour if issue persists
- Catches severe incidents quickly
Slow burn: 3x burn rate over 3 days
- Fires before 30-day budget depletes
- Catches gradual degradation
Alert Fatigue Prevention
Strategies:
- Alert only on actionable issues
- Consolidate related alerts
- Set meaningful thresholds (not arbitrary)
- Require sustained condition before firing
- Include runbook links in every alert
- Review and prune alerts quarterly
Alert quality checklist:
- Can someone take action right now?
- Is the severity appropriate?
- Does it include enough context?
- Is there a runbook linked?
- Has it fired false positives recently?
Dashboard Design
Hierarchy of Dashboards
Service Health Overview:
- High-level SLO status
- Error budget consumption
- Key business metrics
- Designed for quick triage
Deep-Dive Diagnostic:
- Detailed metrics breakdown
- Resource utilization
- Dependency health
- Designed for investigation
Business Metrics:
- User-facing KPIs
- Conversion and engagement
- Revenue impact
- Designed for stakeholders
Dashboard Principles
- Answer specific questions, not show all data
- Use consistent color coding (green=good, red=bad)
- Show time ranges appropriate to the metric
- Include context (deployments, incidents) on graphs
- Mobile-responsive for on-call use
- Provide drill-down paths to detailed views
Essential Panels
| Panel | Purpose | Audience |
|---|---|---|
| SLO Status | Current reliability vs target | Everyone |
| Error Budget | Remaining budget and burn rate | Engineering |
| Request Rate | Traffic patterns and anomalies | Operations |
| Latency Distribution | p50, p95, p99 over time | Engineering |
| Error Breakdown | Errors by type and endpoint | Engineering |
| Dependency Health | Status of upstream services | Operations |
Best Practices
- Correlate metrics, logs, and traces with shared identifiers
- Instrument code at service boundaries, not everywhere
- Use structured logging with consistent field names
- Set retention policies appropriate to data value
- Test alerts in staging before production
- Document SLOs and share with stakeholders
- Conduct regular game days to validate observability
- Automate common diagnostic procedures in runbooks
Anti-Patterns
- Alert on every possible metric (alert fatigue)
- Create dashboards without specific questions in mind
- Log without structure or correlation IDs
- Set SLOs without measuring current baseline
- Ignore error budget policies when convenient
- Treat all alerts with equal severity
- Store high-cardinality data in metrics (use logs/traces)
- Skip postmortems when issues resolve themselves
References
- references/monitoring-patterns.md - Detailed implementation patterns and examples
04 同仓库更多 Skills
seedance-prompt-en
Write effective prompts for Jimeng Seedance 2.0 multimodal AI video generation. Use when users want to create video prompts using text, images, videos, and audio inputs with the @ reference system. Covers camera movements, effects replication, video extension, editing, music beat-matching, e-commerce ads, short dramas, and educational content.
onepage-pdf
Convert an HTML page into a single continuous-page PDF (one tall page, no pagination breaks), preserving desktop layout, backgrounds and selectable text. Use when the user asks to turn an HTML file/report/proposal into a "single-page PDF", "long PDF", "不分页 PDF", "单页 PDF", "长图式 PDF", or complains that an HTML-to-PDF export breaks into pages or cuts content. Supports optional string redaction with leak verification.
native-feel-cross-platform-desktop
Use when the user is designing, prototyping, or rewriting a desktop app that must run on multiple OSes (macOS + Windows, optionally Linux) AND feel indistinguishable from a native app to its users — fast launch, native windowing, native input handling, native materials. Trigger words include "cross-platform desktop", "Electron alternative", "Tauri vs native", "WebView wrapper", "near-native performance", "Raycast architecture", "WebKit/WebView2 quirks", "WKWebView", "system tray app", "global hotkey app", "launcher app". Do NOT trigger this skill for pure web apps, pure mobile apps, or for greenfield projects that have no native-feel requirement.
gstack
Router for the gstack skill suite. (gstack)
fireworks-tech-graph
Use when the user wants to create any technical diagram - architecture, data flow, flowchart, sequence, agent/memory, or concept map - and export as SVG+PNG. Trigger on: "画图" "帮我画" "生成图" "做个图" "架构图" "流程图" "可视化一下" "出图" "generate diagram" "draw diagram" "visualize" or any system/flow description the user wants illustrated.
dbs-video-extract
接收抖音、小红书、微信视频号等短视频链接或分享文案,查询作品/账号数据并提取语音文字稿,按作者和标题归档为 Markdown。用户希望完整解析短视频、取得数据或文字稿时使用;运行前检查 TikHub 与轻抖使用凭证,缺少时用新手能理解的方式引导购买和安全配置。