[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6XG6NPJXHSKXXYRR0DHASE8":3,"skill-same:01M6XG6NPJXHSKXXYRR0DHASE8":65},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":45,"commitSha":46,"skillDirectory":47,"forksCount":14,"repositoryHtmlUrl":48,"repositoryHomepageUrl":6,"ownerHtmlUrl":49,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":50,"install":51,"resources":64},"01M6XG6NPJXHSKXXYRR0DHASE8","distributed-tracing",null,"Implement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.","sapsapshen\u002Fai-memory-router","sapsapshen","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F20347959?v=4","01M6XDQEF1DP50DKF37H81J3J5","shenmian","01M6XDQEJPE41Y3DT2763X8ZJN",0,"MIT","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fcopilot\u002Fdistributed-tracing\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000011","运维与部署","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"distributed","distributed tracing","tracing","implement","microservices","flow","community","across","jaeger","tempo","with","for","unknown","request","2026","2026-11-10 10:12:30",false,"---\nname: distributed-tracing\ndescription: \"Implement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.\"\nrisk: unknown\nsource: community\ndate_added: \"2026-02-27\"\n---\n\n# Distributed Tracing\n\nImplement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.\n\n## Do not use this skill when\n\n- The task is unrelated to distributed tracing\n- You need a different domain or tool outside this scope\n\n## Instructions\n\n- Clarify goals, constraints, and required inputs.\n- Apply relevant best practices and validate outcomes.\n- Provide actionable steps and verification.\n- If detailed examples are required, open `resources\u002Fimplementation-playbook.md`.\n\n## Purpose\n\nTrack requests across distributed systems to understand latency, dependencies, and failure points.\n\n## Use this skill when\n\n- Debug latency issues\n- Understand service dependencies\n- Identify bottlenecks\n- Trace error propagation\n- Analyze request paths\n\n## Distributed Tracing Concepts\n\n### Trace Structure\n```\nTrace (Request ID: abc123)\n  ↓\nSpan (frontend) [100ms]\n  ↓\nSpan (api-gateway) [80ms]\n  ├→ Span (auth-service) [10ms]\n  └→ Span (user-service) [60ms]\n      └→ Span (database) [40ms]\n```\n\n### Key Components\n- **Trace** - End-to-end request journey\n- **Span** - Single operation within a trace\n- **Context** - Metadata propagated between services\n- **Tags** - Key-value pairs for filtering\n- **Logs** - Timestamped events within a span\n\n## Jaeger Setup\n\n### Kubernetes Deployment\n\n```bash\n# Deploy Jaeger Operator\nkubectl create namespace observability\nkubectl create -f https:\u002F\u002Fgithub.com\u002Fjaegertracing\u002Fjaeger-operator\u002Freleases\u002Fdownload\u002Fv1.51.0\u002Fjaeger-operator.yaml -n observability\n\n# Deploy Jaeger instance\nkubectl apply -f - \u003C\u003CEOF\napiVersion: jaegertracing.io\u002Fv1\nkind: Jaeger\nmetadata:\n  name: jaeger\n  namespace: observability\nspec:\n  strategy: production\n  storage:\n    type: elasticsearch\n    options:\n      es:\n        server-urls: http:\u002F\u002Felasticsearch:9200\n  ingress:\n    enabled: true\nEOF\n```\n\n### Docker Compose\n\n```yaml\nversion: '3.8'\nservices:\n  jaeger:\n    image: jaegertracing\u002Fall-in-one:latest\n    ports:\n      - \"5775:5775\u002Fudp\"\n      - \"6831:6831\u002Fudp\"\n      - \"6832:6832\u002Fudp\"\n      - \"5778:5778\"\n      - \"16686:16686\"  # UI\n      - \"14268:14268\"  # Collector\n      - \"14250:14250\"  # gRPC\n      - \"9411:9411\"    # Zipkin\n    environment:\n      - COLLECTOR_ZIPKIN_HOST_PORT=:9411\n```\n\n**Reference:** See `references\u002Fjaeger-setup.md`\n\n## Application Instrumentation\n\n### OpenTelemetry (Recommended)\n\n#### Python (Flask)\n```python\nfrom opentelemetry import trace\nfrom opentelemetry.exporter.jaeger.thrift import JaegerExporter\nfrom opentelemetry.sdk.resources import SERVICE_NAME, Resource\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nfrom opentelemetry.instrumentation.flask import FlaskInstrumentor\nfrom flask import Flask\n\n# Initialize tracer\nresource = Resource(attributes={SERVICE_NAME: \"my-service\"})\nprovider = TracerProvider(resource=resource)\nprocessor = BatchSpanProcessor(JaegerExporter(\n    agent_host_name=\"jaeger\",\n    agent_port=6831,\n))\nprovider.add_span_processor(processor)\ntrace.set_tracer_provider(provider)\n\n# Instrument Flask\napp = Flask(__name__)\nFlaskInstrumentor().instrument_app(app)\n\n@app.route('\u002Fapi\u002Fusers')\ndef get_users():\n    tracer = trace.get_tracer(__name__)\n\n    with tracer.start_as_current_span(\"get_users\") as span:\n        span.set_attribute(\"user.count\", 100)\n        # Business logic\n        users = fetch_users_from_db()\n        return {\"users\": users}\n\ndef fetch_users_from_db():\n    tracer = trace.get_tracer(__name__)\n\n    with tracer.start_as_current_span(\"database_query\") as span:\n        span.set_attribute(\"db.system\", \"postgresql\")\n        span.set_attribute(\"db.statement\", \"SELECT * FROM users\")\n        # Database query\n        return query_database()\n```\n\n#### Node.js (Express)\n```javascript\nconst { NodeTracerProvider } = require('@opentelemetry\u002Fsdk-trace-node');\nconst { JaegerExporter } = require('@opentelemetry\u002Fexporter-jaeger');\nconst { BatchSpanProcessor } = require('@opentelemetry\u002Fsdk-trace-base');\nconst { registerInstrumentations } = require('@opentelemetry\u002Finstrumentation');\nconst { HttpInstrumentation } = require('@opentelemetry\u002Finstrumentation-http');\nconst { ExpressInstrumentation } = require('@opentelemetry\u002Finstrumentation-express');\n\n\u002F\u002F Initialize tracer\nconst provider = new NodeTracerProvider({\n  resource: { attributes: { 'service.name': 'my-service' } }\n});\n\nconst exporter = new JaegerExporter({\n  endpoint: 'http:\u002F\u002Fjaeger:14268\u002Fapi\u002Ftraces'\n});\n\nprovider.addSpanProcessor(new BatchSpanProcessor(exporter));\nprovider.register();\n\n\u002F\u002F Instrument libraries\nregisterInstrumentations({\n  instrumentations: [\n    new HttpInstrumentation(),\n    new ExpressInstrumentation(),\n  ],\n});\n\nconst express = require('express');\nconst app = express();\n\napp.get('\u002Fapi\u002Fusers', async (req, res) => {\n  const tracer = trace.getTracer('my-service');\n  const span = tracer.startSpan('get_users');\n\n  try {\n    const users = await fetchUsers();\n    span.setAttributes({ 'user.count': users.length });\n    res.json({ users });\n  } finally {\n    span.end();\n  }\n});\n```\n\n#### Go\n```go\npackage main\n\nimport (\n    \"context\"\n    \"go.opentelemetry.io\u002Fotel\"\n    \"go.opentelemetry.io\u002Fotel\u002Fexporters\u002Fjaeger\"\n    \"go.opentelemetry.io\u002Fotel\u002Fsdk\u002Fresource\"\n    sdktrace \"go.opentelemetry.io\u002Fotel\u002Fsdk\u002Ftrace\"\n    semconv \"go.opentelemetry.io\u002Fotel\u002Fsemconv\u002Fv1.4.0\"\n)\n\nfunc initTracer() (*sdktrace.TracerProvider, error) {\n    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(\n        jaeger.WithEndpoint(\"http:\u002F\u002Fjaeger:14268\u002Fapi\u002Ftraces\"),\n    ))\n    if err != nil {\n        return nil, err\n    }\n\n    tp := sdktrace.NewTracerProvider(\n        sdktrace.WithBatcher(exporter),\n        sdktrace.WithResource(resource.NewWithAttributes(\n            semconv.SchemaURL,\n            semconv.ServiceNameKey.String(\"my-service\"),\n        )),\n    )\n\n    otel.SetTracerProvider(tp)\n    return tp, nil\n}\n\nfunc getUsers(ctx context.Context) ([]User, error) {\n    tracer := otel.Tracer(\"my-service\")\n    ctx, span := tracer.Start(ctx, \"get_users\")\n    defer span.End()\n\n    span.SetAttributes(attribute.String(\"user.filter\", \"active\"))\n\n    users, err := fetchUsersFromDB(ctx)\n    if err != nil {\n        span.RecordError(err)\n        return nil, err\n    }\n\n    span.SetAttributes(attribute.Int(\"user.count\", len(users)))\n    return users, nil\n}\n```\n\n**Reference:** See `references\u002Finstrumentation.md`\n\n## Context Propagation\n\n### HTTP Headers\n```\ntraceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01\ntracestate: congo=t61rcWkgMzE\n```\n\n### Propagation in HTTP Requests\n\n#### Python\n```python\nfrom opentelemetry.propagate import inject\n\nheaders = {}\ninject(headers)  # Injects trace context\n\nresponse = requests.get('http:\u002F\u002Fdownstream-service\u002Fapi', headers=headers)\n```\n\n#### Node.js\n```javascript\nconst { propagation } = require('@opentelemetry\u002Fapi');\n\nconst headers = {};\npropagation.inject(context.active(), headers);\n\naxios.get('http:\u002F\u002Fdownstream-service\u002Fapi', { headers });\n```\n\n## Tempo Setup (Grafana)\n\n### Kubernetes Deployment\n\n```yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: tempo-config\ndata:\n  tempo.yaml: |\n    server:\n      http_listen_port: 3200\n\n    distributor:\n      receivers:\n        jaeger:\n          protocols:\n            thrift_http:\n            grpc:\n        otlp:\n          protocols:\n            http:\n            grpc:\n\n    storage:\n      trace:\n        backend: s3\n        s3:\n          bucket: tempo-traces\n          endpoint: s3.amazonaws.com\n\n    querier:\n      frontend_worker:\n        frontend_address: tempo-query-frontend:9095\n---\napiVersion: apps\u002Fv1\nkind: Deployment\nmetadata:\n  name: tempo\nspec:\n  replicas: 1\n  template:\n    spec:\n      containers:\n      - name: tempo\n        image: grafana\u002Ftempo:latest\n        args:\n          - -config.file=\u002Fetc\u002Ftempo\u002Ftempo.yaml\n        volumeMounts:\n        - name: config\n          mountPath: \u002Fetc\u002Ftempo\n      volumes:\n      - name: config\n        configMap:\n          name: tempo-config\n```\n\n**Reference:** See `assets\u002Fjaeger-config.yaml.template`\n\n## Sampling Strategies\n\n### Probabilistic Sampling\n```yaml\n# Sample 1% of traces\nsampler:\n  type: probabilistic\n  param: 0.01\n```\n\n### Rate Limiting Sampling\n```yaml\n# Sample max 100 traces per second\nsampler:\n  type: ratelimiting\n  param: 100\n```\n\n### Adaptive Sampling\n```python\nfrom opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased\n\n# Sample based on trace ID (deterministic)\nsampler = ParentBased(root=TraceIdRatioBased(0.01))\n```\n\n## Trace Analysis\n\n### Finding Slow Requests\n\n**Jaeger Query:**\n```\nservice=my-service\nduration > 1s\n```\n\n### Finding Errors\n\n**Jaeger Query:**\n```\nservice=my-service\nerror=true\ntags.http.status_code >= 500\n```\n\n### Service Dependency Graph\n\nJaeger automatically generates service dependency graphs showing:\n- Service relationships\n- Request rates\n- Error rates\n- Average latencies\n\n## Best Practices\n\n1. **Sample appropriately** (1-10% in production)\n2. **Add meaningful tags** (user_id, request_id)\n3. **Propagate context** across all service boundaries\n4. **Log exceptions** in spans\n5. **Use consistent naming** for operations\n6. **Monitor tracing overhead** (\u003C1% CPU impact)\n7. **Set up alerts** for trace errors\n8. **Implement distributed context** (baggage)\n9. **Use span events** for important milestones\n10. **Document instrumentation** standards\n\n## Integration with Logging\n\n### Correlated Logs\n```python\nimport logging\nfrom opentelemetry import trace\n\nlogger = logging.getLogger(__name__)\n\ndef process_request():\n    span = trace.get_current_span()\n    trace_id = span.get_span_context().trace_id\n\n    logger.info(\n        \"Processing request\",\n        extra={\"trace_id\": format(trace_id, '032x')}\n    )\n```\n\n## Troubleshooting\n\n**No traces appearing:**\n- Check collector endpoint\n- Verify network connectivity\n- Check sampling configuration\n- Review application logs\n\n**High latency overhead:**\n- Reduce sampling rate\n- Use batch span processor\n- Check exporter configuration\n\n## Reference Files\n\n- `references\u002Fjaeger-setup.md` - Jaeger installation\n- `references\u002Finstrumentation.md` - Instrumentation patterns\n- `assets\u002Fjaeger-config.yaml.template` - Jaeger configuration\n\n## Related Skills\n\n- `prometheus-configuration` - For metrics\n- `grafana-dashboards` - For visualization\n- `slo-implementation` - For latency SLOs\n","# Distributed Tracing\n\nImplement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.\n\n## Do not use this skill when\n\n- The task is unrelated to distributed tracing\n- You need a different domain or tool outside this scope\n\n## Instructions\n\n- Clarify goals, constraints, and required inputs.\n- Apply relevant best practices and validate outcomes.\n- Provide actionable steps and verification.\n- If detailed examples are required, open `resources\u002Fimplementation-playbook.md`.\n\n## Purpose\n\nTrack requests across distributed systems to understand latency, dependencies, and failure points.\n\n## Use this skill when\n\n- Debug latency issues\n- Understand service dependencies\n- Identify bottlenecks\n- Trace error propagation\n- Analyze request paths\n\n## Distributed Tracing Concepts\n\n### Trace Structure\n```\nTrace (Request ID: abc123)\n  ↓\nSpan (frontend) [100ms]\n  ↓\nSpan (api-gateway) [80ms]\n  ├→ Span (auth-service) [10ms]\n  └→ Span (user-service) [60ms]\n      └→ Span (database) [40ms]\n```\n\n### Key Components\n- **Trace** - End-to-end request journey\n- **Span** - Single operation within a trace\n- **Context** - Metadata propagated between services\n- **Tags** - Key-value pairs for filtering\n- **Logs** - Timestamped events within a span\n\n## Jaeger Setup\n\n### Kubernetes Deployment\n\n```bash\n# Deploy Jaeger Operator\nkubectl create namespace observability\nkubectl create -f https:\u002F\u002Fgithub.com\u002Fjaegertracing\u002Fjaeger-operator\u002Freleases\u002Fdownload\u002Fv1.51.0\u002Fjaeger-operator.yaml -n observability\n\n# Deploy Jaeger instance\nkubectl apply -f - \u003C\u003CEOF\napiVersion: jaegertracing.io\u002Fv1\nkind: Jaeger\nmetadata:\n  name: jaeger\n  namespace: observability\nspec:\n  strategy: production\n  storage:\n    type: elasticsearch\n    options:\n      es:\n        server-urls: http:\u002F\u002Felasticsearch:9200\n  ingress:\n    enabled: true\nEOF\n```\n\n### Docker Compose\n\n```yaml\nversion: '3.8'\nservices:\n  jaeger:\n    image: jaegertracing\u002Fall-in-one:latest\n    ports:\n      - \"5775:5775\u002Fudp\"\n      - \"6831:6831\u002Fudp\"\n      - \"6832:6832\u002Fudp\"\n      - \"5778:5778\"\n      - \"16686:16686\"  # UI\n      - \"14268:14268\"  # Collector\n      - \"14250:14250\"  # gRPC\n      - \"9411:9411\"    # Zipkin\n    environment:\n      - COLLECTOR_ZIPKIN_HOST_PORT=:9411\n```\n\n**Reference:** See `references\u002Fjaeger-setup.md`\n\n## Application Instrumentation\n\n### OpenTelemetry (Recommended)\n\n#### Python (Flask)\n```python\nfrom opentelemetry import trace\nfrom opentelemetry.exporter.jaeger.thrift import JaegerExporter\nfrom opentelemetry.sdk.resources import SERVICE_NAME, Resource\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nfrom opentelemetry.instrumentation.flask import FlaskInstrumentor\nfrom flask import Flask\n\n# Initialize tracer\nresource = Resource(attributes={SERVICE_NAME: \"my-service\"})\nprovider = TracerProvider(resource=resource)\nprocessor = BatchSpanProcessor(JaegerExporter(\n    agent_host_name=\"jaeger\",\n    agent_port=6831,\n))\nprovider.add_span_processor(processor)\ntrace.set_tracer_provider(provider)\n\n# Instrument Flask\napp = Flask(__name__)\nFlaskInstrumentor().instrument_app(app)\n\n@app.route('\u002Fapi\u002Fusers')\ndef get_users():\n    tracer = trace.get_tracer(__name__)\n\n    with tracer.start_as_current_span(\"get_users\") as span:\n        span.set_attribute(\"user.count\", 100)\n        # Business logic\n        users = fetch_users_from_db()\n        return {\"users\": users}\n\ndef fetch_users_from_db():\n    tracer = trace.get_tracer(__name__)\n\n    with tracer.start_as_current_span(\"database_query\") as span:\n        span.set_attribute(\"db.system\", \"postgresql\")\n        span.set_attribute(\"db.statement\", \"SELECT * FROM users\")\n        # Database query\n        return query_database()\n```\n\n#### Node.js (Express)\n```javascript\nconst { NodeTracerProvider } = require('@opentelemetry\u002Fsdk-trace-node');\nconst { JaegerExporter } = require('@opentelemetry\u002Fexporter-jaeger');\nconst { BatchSpanProcessor } = require('@opentelemetry\u002Fsdk-trace-base');\nconst { registerInstrumentations } = require('@opentelemetry\u002Finstrumentation');\nconst { HttpInstrumentation } = require('@opentelemetry\u002Finstrumentation-http');\nconst { ExpressInstrumentation } = require('@opentelemetry\u002Finstrumentation-express');\n\n\u002F\u002F Initialize tracer\nconst provider = new NodeTracerProvider({\n  resource: { attributes: { 'service.name': 'my-service' } }\n});\n\nconst exporter = new JaegerExporter({\n  endpoint: 'http:\u002F\u002Fjaeger:14268\u002Fapi\u002Ftraces'\n});\n\nprovider.addSpanProcessor(new BatchSpanProcessor(exporter));\nprovider.register();\n\n\u002F\u002F Instrument libraries\nregisterInstrumentations({\n  instrumentations: [\n    new HttpInstrumentation(),\n    new ExpressInstrumentation(),\n  ],\n});\n\nconst express = require('express');\nconst app = express();\n\napp.get('\u002Fapi\u002Fusers', async (req, res) => {\n  const tracer = trace.getTracer('my-service');\n  const span = tracer.startSpan('get_users');\n\n  try {\n    const users = await fetchUsers();\n    span.setAttributes({ 'user.count': users.length });\n    res.json({ users });\n  } finally {\n    span.end();\n  }\n});\n```\n\n#### Go\n```go\npackage main\n\nimport (\n    \"context\"\n    \"go.opentelemetry.io\u002Fotel\"\n    \"go.opentelemetry.io\u002Fotel\u002Fexporters\u002Fjaeger\"\n    \"go.opentelemetry.io\u002Fotel\u002Fsdk\u002Fresource\"\n    sdktrace \"go.opentelemetry.io\u002Fotel\u002Fsdk\u002Ftrace\"\n    semconv \"go.opentelemetry.io\u002Fotel\u002Fsemconv\u002Fv1.4.0\"\n)\n\nfunc initTracer() (*sdktrace.TracerProvider, error) {\n    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(\n        jaeger.WithEndpoint(\"http:\u002F\u002Fjaeger:14268\u002Fapi\u002Ftraces\"),\n    ))\n    if err != nil {\n        return nil, err\n    }\n\n    tp := sdktrace.NewTracerProvider(\n        sdktrace.WithBatcher(exporter),\n        sdktrace.WithResource(resource.NewWithAttributes(\n            semconv.SchemaURL,\n            semconv.ServiceNameKey.String(\"my-service\"),\n        )),\n    )\n\n    otel.SetTracerProvider(tp)\n    return tp, nil\n}\n\nfunc getUsers(ctx context.Context) ([]User, error) {\n    tracer := otel.Tracer(\"my-service\")\n    ctx, span := tracer.Start(ctx, \"get_users\")\n    defer span.End()\n\n    span.SetAttributes(attribute.String(\"user.filter\", \"active\"))\n\n    users, err := fetchUsersFromDB(ctx)\n    if err != nil {\n        span.RecordError(err)\n        return nil, err\n    }\n\n    span.SetAttributes(attribute.Int(\"user.count\", len(users)))\n    return users, nil\n}\n```\n\n**Reference:** See `references\u002Finstrumentation.md`\n\n## Context Propagation\n\n### HTTP Headers\n```\ntraceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01\ntracestate: congo=t61rcWkgMzE\n```\n\n### Propagation in HTTP Requests\n\n#### Python\n```python\nfrom opentelemetry.propagate import inject\n\nheaders = {}\ninject(headers)  # Injects trace context\n\nresponse = requests.get('http:\u002F\u002Fdownstream-service\u002Fapi', headers=headers)\n```\n\n#### Node.js\n```javascript\nconst { propagation } = require('@opentelemetry\u002Fapi');\n\nconst headers = {};\npropagation.inject(context.active(), headers);\n\naxios.get('http:\u002F\u002Fdownstream-service\u002Fapi', { headers });\n```\n\n## Tempo Setup (Grafana)\n\n### Kubernetes Deployment\n\n```yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: tempo-config\ndata:\n  tempo.yaml: |\n    server:\n      http_listen_port: 3200\n\n    distributor:\n      receivers:\n        jaeger:\n          protocols:\n            thrift_http:\n            grpc:\n        otlp:\n          protocols:\n            http:\n            grpc:\n\n    storage:\n      trace:\n        backend: s3\n        s3:\n          bucket: tempo-traces\n          endpoint: s3.amazonaws.com\n\n    querier:\n      frontend_worker:\n        frontend_address: tempo-query-frontend:9095\n---\napiVersion: apps\u002Fv1\nkind: Deployment\nmetadata:\n  name: tempo\nspec:\n  replicas: 1\n  template:\n    spec:\n      containers:\n      - name: tempo\n        image: grafana\u002Ftempo:latest\n        args:\n          - -config.file=\u002Fetc\u002Ftempo\u002Ftempo.yaml\n        volumeMounts:\n        - name: config\n          mountPath: \u002Fetc\u002Ftempo\n      volumes:\n      - name: config\n        configMap:\n          name: tempo-config\n```\n\n**Reference:** See `assets\u002Fjaeger-config.yaml.template`\n\n## Sampling Strategies\n\n### Probabilistic Sampling\n```yaml\n# Sample 1% of traces\nsampler:\n  type: probabilistic\n  param: 0.01\n```\n\n### Rate Limiting Sampling\n```yaml\n# Sample max 100 traces per second\nsampler:\n  type: ratelimiting\n  param: 100\n```\n\n### Adaptive Sampling\n```python\nfrom opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased\n\n# Sample based on trace ID (deterministic)\nsampler = ParentBased(root=TraceIdRatioBased(0.01))\n```\n\n## Trace Analysis\n\n### Finding Slow Requests\n\n**Jaeger Query:**\n```\nservice=my-service\nduration > 1s\n```\n\n### Finding Errors\n\n**Jaeger Query:**\n```\nservice=my-service\nerror=true\ntags.http.status_code >= 500\n```\n\n### Service Dependency Graph\n\nJaeger automatically generates service dependency graphs showing:\n- Service relationships\n- Request rates\n- Error rates\n- Average latencies\n\n## Best Practices\n\n1. **Sample appropriately** (1-10% in production)\n2. **Add meaningful tags** (user_id, request_id)\n3. **Propagate context** across all service boundaries\n4. **Log exceptions** in spans\n5. **Use consistent naming** for operations\n6. **Monitor tracing overhead** (\u003C1% CPU impact)\n7. **Set up alerts** for trace errors\n8. **Implement distributed context** (baggage)\n9. **Use span events** for important milestones\n10. **Document instrumentation** standards\n\n## Integration with Logging\n\n### Correlated Logs\n```python\nimport logging\nfrom opentelemetry import trace\n\nlogger = logging.getLogger(__name__)\n\ndef process_request():\n    span = trace.get_current_span()\n    trace_id = span.get_span_context().trace_id\n\n    logger.info(\n        \"Processing request\",\n        extra={\"trace_id\": format(trace_id, '032x')}\n    )\n```\n\n## Troubleshooting\n\n**No traces appearing:**\n- Check collector endpoint\n- Verify network connectivity\n- Check sampling configuration\n- Review application logs\n\n**High latency overhead:**\n- Reduce sampling rate\n- Use batch span processor\n- Check exporter configuration\n\n## Reference Files\n\n- `references\u002Fjaeger-setup.md` - Jaeger installation\n- `references\u002Finstrumentation.md` - Instrumentation patterns\n- `assets\u002Fjaeger-config.yaml.template` - Jaeger configuration\n\n## Related Skills\n\n- `prometheus-configuration` - For metrics\n- `grafana-dashboards` - For visualization\n- `slo-implementation` - For latency SLOs\n",{"name":5,"risk":36,"source":30,"date_added":44,"description":7},"2026-02-27","main","7a38d8e7073e61c16eda85681cb32fb68a8657a5","skills-store\u002Fcopilot\u002Fdistributed-tracing","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router","https:\u002F\u002Fgithub.com\u002Fsapsapshen","2026-04-14 21:12:12",{"source":8,"skillName":5,"commands":52,"prompt":62,"zipAvailable":63},[53,56,59],{"tool":54,"command":55},"npx","npx skills add sapsapshen\u002Fai-memory-router --skill distributed-tracing",{"tool":57,"command":58},"bunx","bunx skills add sapsapshen\u002Fai-memory-router --skill distributed-tracing",{"tool":60,"command":61},"pnpm","pnpm dlx skills add sapsapshen\u002Fai-memory-router --skill distributed-tracing","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fcopilot\u002Fdistributed-tracing\u002FSKILL.md",true,[],[66,89,114,124,137,154],{"skillNo":67,"name":68,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":77,"publishedAt":39},"01M71ATQRKG48SH06FQ697T5G9","wechat-search","Search WeChat Official Account articles using OpenClaw's web search and fetch capabilities with compliance-focused design.","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fopenclaw\u002Fwechat-search-release\u002FSKILL.md",[72],{"categoryNo":73,"categoryName":74,"taxonomyType":75,"taxonomyTypeDesc":76},"01JBSEEDCAT000000000000037","律师与法务","PROFESSION","适用职业",[78,79,80,81,82,83,84,85,86,87,88,34],"wechat search","wechat","search","capabilities","openclaw","design","web search","compliance","web","official","fetch",{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":93,"categories":94,"keywords":95,"publishedAt":39},"01M71ATNAVGZB35PFYM2GC5N00","trend-tap","Real-time trending topics aggregator across 7 platforms (X\u002FTwitter, Reddit, Google Trends, Hacker News, Zhihu, Bilibili, Weibo). Trigger: when user says 'trend', 'trends', 'trend tap', '热点', '热搜', or asks what is trending\u002Fhot. Do NOT use web_search — this skill already fetches live data from all platforms with zero API keys.","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fopenclaw\u002Ftrend-tap\u002FSKILL.md",[],[96,97,98,99,100,101,80,102,103,104,105,106,107,108,109,110,111,112,31,113],"trend","trend tap","tap","api keys","Bash","time","trending","hacker","aggregator","do","when","brew","live","data","google","install","read","api",{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":39},"01M71ATJ9AM5KM15AA8PS302EB","trend-radar","Real-time trending topics aggregator across 7 platforms (X\u002FTwitter, Reddit, Google Trends, Hacker News, Zhihu, Bilibili, Weibo). Trigger: when user says 'trend', 'trends', 'trend pulse', '热点', '热搜', or asks what is trending\u002Fhot. Do NOT use web_search — this skill already fetches live data from all platforms with zero API keys.","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fopenclaw\u002Ftrend-radar\u002FSKILL.md",[],[121,122,96,107,108,109,110,100,80,112,123,31,113,106,104,103,102,111,101,99],"trend radar","radar","pulse",{"skillNo":125,"name":126,"nameZh":6,"description":127,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":128,"categories":129,"keywords":130,"publishedAt":39},"01M71ATGF095N3EJM9G8SD0BB5","tech-news-daily","每日科技新闻汇——从全球多个 RSS 源抓取最新科技新闻，翻译（英→中）、去重、排版成每日简报。当用户询问科技新闻、科技新鲜事、每日科技、技术动态、Daily Tech News 或类似表达时触发此技能。","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fopenclaw\u002Ftech-news-daily\u002FSKILL.md",[],[131,132,133,134,135,136],"news","tech","tech news daily","daily","rss","翻译",{"skillNo":138,"name":139,"nameZh":6,"description":140,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":141,"categories":142,"keywords":146,"publishedAt":39},"01M71ATFAEAMQ17M70NQNY7SR8","summarize","Summarize URLs or files with the summarize CLI (web, PDFs, images, audio, YouTube).","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fopenclaw\u002Fsummarize\u002FSKILL.md",[143],{"categoryNo":144,"categoryName":145,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000021","资料检索与研究",[139,86,147,148,149,34,107,150,151,152,98,153,111],"Audio","cli","youtube","images","https","files","pdfs",{"skillNo":155,"name":156,"nameZh":6,"description":157,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":158,"categories":159,"keywords":160,"publishedAt":39},"01M71ATDNS3R0M4H936PACQAWZ","sonoscli","Control Sonos speakers (discover\u002Fstatus\u002Fplay\u002Fvolume\u002Fgroup).","https:\u002F\u002Fgithub.com\u002Fsapsapshen\u002Fai-memory-router\u002Fblob\u002Fmain\u002Fskills-store\u002Fopenclaw\u002Fsonoscli\u002FSKILL.md",[],[156,161,162,163,111,164,151,165,166,167,168,169],"status","play","go","latest","github","discover","control","cmd","volume"]