AGENT SKILL

observability-design

Monitoring strategies, distributed tracing, SLI/SLO design, and alerting patterns. Use when designing monitoring infrastructure, defining service level objectives, implementing distributed tracing, creating alert rules, building dashboards, or establishing incident response procedures. Covers the three pillars of observability and production readiness.

适用职业设计师
使用场景视觉设计
License 许可
MIT
最近更新
2026-11-01
Stars
★ 3
Forks
1

01 前言 Front Matter

name
observability-design
description
Monitoring strategies, distributed tracing, SLI/SLO design, and alerting patterns. Use when designing monitoring infrastructure, defining service level objectives, implementing distributed tracing, creating alert rules, building dashboards, or establishing incident response procedures. Covers the three pillars of observability and production readiness.

02 资源文件 1

03 SKILL.md 正文

Observability Patterns

When to Use

  • Designing monitoring infrastructure for new services
  • Defining SLIs, SLOs, and error budgets for reliability
  • Implementing distributed tracing across microservices
  • Creating alert rules that minimize noise and maximize signal
  • Building dashboards for operations and business stakeholders
  • Establishing incident response and postmortem processes
  • Diagnosing production issues through telemetry correlation

Philosophy

You cannot fix what you cannot see. Observability is not about collecting data - it is about answering questions you have not thought to ask yet. Good observability turns every incident into a learning opportunity and every metric into actionable insight.

The Three Pillars

Metrics

Numeric measurements aggregated over time. Best for understanding system behavior at scale.

Characteristics:

  • Highly efficient storage (aggregated values)
  • Support mathematical operations (rates, percentiles)
  • Enable alerting on thresholds
  • Limited cardinality (avoid high-cardinality labels)

Types:

Type Use Case Example
Counter Cumulative values that only increase Total requests, errors, bytes sent
Gauge Values that go up and down Current memory, active connections
Histogram Distribution of values in buckets Request latency, payload sizes
Summary Similar to histogram, calculated client-side Pre-computed percentiles

Logs

Immutable records of discrete events. Best for understanding specific occurrences.

Characteristics:

  • Rich context and arbitrary data
  • Expensive to store and query at scale
  • Essential for debugging specific issues
  • Should be structured (JSON) for parseability

Structure:

Required fields:
- timestamp: ISO 8601 format with timezone
- level: ERROR, WARN, INFO, DEBUG
- message: Human-readable description
- service: Service identifier
- trace_id: Correlation identifier

Context fields:
- user_id: Sanitized user identifier
- request_id: Request correlation
- duration_ms: Operation timing
- error_type: Classification for errors

Traces

Records of request flow across distributed systems. Best for understanding causality and latency.

Characteristics:

  • Show request path through services
  • Identify latency bottlenecks
  • Reveal dependencies and failure points
  • Higher overhead than metrics

Components:

  • Trace: Complete request journey
  • Span: Single operation within a trace
  • Context: Metadata propagated across services

SLI/SLO/SLA Framework

Service Level Indicators (SLIs)

Quantitative measures of service behavior from the user perspective.

Common SLI categories:

Category Measures Example SLI
Availability Service is responding % of successful requests
Latency Response speed % of requests < 200ms
Throughput Capacity Requests processed per second
Error Rate Correctness % of requests without errors
Freshness Data currency % of data < 1 minute old

SLI specification:

SLI: Request Latency
Definition: Time from request received to response sent
Measurement: Server-side histogram at p50, p95, p99
Exclusions: Health checks, internal tooling
Data source: Application metrics

Service Level Objectives (SLOs)

Target reliability levels for SLIs over a time window.

SLO formula:

SLO = (Good events / Total events) >= Target over Window

Example:
99.9% of requests complete successfully in < 200ms
measured over a 30-day rolling window

Setting SLO targets:

  • Start with current baseline performance
  • Consider user expectations and business impact
  • Balance reliability investment against feature velocity
  • Document error budget policy

Error Budgets

The allowed amount of unreliability within an SLO.

Calculation:

Error Budget = 1 - SLO Target

99.9% SLO = 0.1% error budget
= 43.2 minutes downtime per 30 days
= 8.64 seconds per day

Error budget policies:

  • Budget remaining: Continue feature development
  • Budget depleted: Focus on reliability work
  • Budget burning fast: Freeze deploys, investigate

Alerting Strategies

Symptom-Based Alerts

Alert on user-visible symptoms, not internal causes.

Good alerts:

  • Error rate exceeds threshold (users experiencing failures)
  • Latency SLO at risk (users experiencing slowness)
  • Queue depth growing (backlog affecting users)

Poor alerts:

  • CPU at 80% (may not affect users)
  • Pod restarted (self-healing, may not affect users)
  • Disk at 70% (not yet impacting service)

Multi-Window, Multi-Burn-Rate Alerts

Detect fast burns quickly, slow burns before budget depletion.

Configuration:

Fast burn: 14.4x burn rate over 1 hour
  - Fires in 1 hour if issue persists
  - Catches severe incidents quickly

Slow burn: 3x burn rate over 3 days
  - Fires before 30-day budget depletes
  - Catches gradual degradation

Alert Fatigue Prevention

Strategies:

  • Alert only on actionable issues
  • Consolidate related alerts
  • Set meaningful thresholds (not arbitrary)
  • Require sustained condition before firing
  • Include runbook links in every alert
  • Review and prune alerts quarterly

Alert quality checklist:

  • Can someone take action right now?
  • Is the severity appropriate?
  • Does it include enough context?
  • Is there a runbook linked?
  • Has it fired false positives recently?

Dashboard Design

Hierarchy of Dashboards

Service Health Overview:

  • High-level SLO status
  • Error budget consumption
  • Key business metrics
  • Designed for quick triage

Deep-Dive Diagnostic:

  • Detailed metrics breakdown
  • Resource utilization
  • Dependency health
  • Designed for investigation

Business Metrics:

  • User-facing KPIs
  • Conversion and engagement
  • Revenue impact
  • Designed for stakeholders

Dashboard Principles

  • Answer specific questions, not show all data
  • Use consistent color coding (green=good, red=bad)
  • Show time ranges appropriate to the metric
  • Include context (deployments, incidents) on graphs
  • Mobile-responsive for on-call use
  • Provide drill-down paths to detailed views

Essential Panels

Panel Purpose Audience
SLO Status Current reliability vs target Everyone
Error Budget Remaining budget and burn rate Engineering
Request Rate Traffic patterns and anomalies Operations
Latency Distribution p50, p95, p99 over time Engineering
Error Breakdown Errors by type and endpoint Engineering
Dependency Health Status of upstream services Operations

Best Practices

  • Correlate metrics, logs, and traces with shared identifiers
  • Instrument code at service boundaries, not everywhere
  • Use structured logging with consistent field names
  • Set retention policies appropriate to data value
  • Test alerts in staging before production
  • Document SLOs and share with stakeholders
  • Conduct regular game days to validate observability
  • Automate common diagnostic procedures in runbooks

Anti-Patterns

  • Alert on every possible metric (alert fatigue)
  • Create dashboards without specific questions in mind
  • Log without structure or correlation IDs
  • Set SLOs without measuring current baseline
  • Ignore error budget policies when convenient
  • Treat all alerts with equal severity
  • Store high-cardinality data in metrics (use logs/traces)
  • Skip postmortems when issues resolve themselves

References

04 同仓库更多 Skills

shengdabai
shengdabai/Tony-Claude-Code-Skills

seedance-prompt-en

Write effective prompts for Jimeng Seedance 2.0 multimodal AI video generation. Use when users want to create video prompts using text, images, videos, and audio inputs with the @ reference system. Covers camera movements, effects replication, video extension, editing, music beat-matching, e-commerce ads, short dramas, and educational content.

★ 3
shengdabai
shengdabai/Tony-Claude-Code-Skills

onepage-pdf

Convert an HTML page into a single continuous-page PDF (one tall page, no pagination breaks), preserving desktop layout, backgrounds and selectable text. Use when the user asks to turn an HTML file/report/proposal into a "single-page PDF", "long PDF", "不分页 PDF", "单页 PDF", "长图式 PDF", or complains that an HTML-to-PDF export breaks into pages or cuts content. Supports optional string redaction with leak verification.

★ 3
shengdabai
shengdabai/Tony-Claude-Code-Skills

native-feel-cross-platform-desktop

Use when the user is designing, prototyping, or rewriting a desktop app that must run on multiple OSes (macOS + Windows, optionally Linux) AND feel indistinguishable from a native app to its users — fast launch, native windowing, native input handling, native materials. Trigger words include "cross-platform desktop", "Electron alternative", "Tauri vs native", "WebView wrapper", "near-native performance", "Raycast architecture", "WebKit/WebView2 quirks", "WKWebView", "system tray app", "global hotkey app", "launcher app". Do NOT trigger this skill for pure web apps, pure mobile apps, or for greenfield projects that have no native-feel requirement.

★ 3
shengdabai
shengdabai/Tony-Claude-Code-Skills

gstack

Router for the gstack skill suite. (gstack)

★ 3
shengdabai
shengdabai/Tony-Claude-Code-Skills

fireworks-tech-graph

Use when the user wants to create any technical diagram - architecture, data flow, flowchart, sequence, agent/memory, or concept map - and export as SVG+PNG. Trigger on: "画图" "帮我画" "生成图" "做个图" "架构图" "流程图" "可视化一下" "出图" "generate diagram" "draw diagram" "visualize" or any system/flow description the user wants illustrated.

★ 3
shengdabai
shengdabai/Tony-Claude-Code-Skills

dbs-video-extract

接收抖音、小红书、微信视频号等短视频链接或分享文案,查询作品/账号数据并提取语音文字稿,按作者和标题归档为 Markdown。用户希望完整解析短视频、取得数据或文字稿时使用;运行前检查 TikHub 与轻抖使用凭证,缺少时用新手能理解的方式引导购买和安全配置。

★ 3