cloud-monitoring-alert-configuration

Configure Cloud Monitoring Alerts

Configure high-signal alerting policies in Cloud Monitoring using Terraform (google_monitoring_alert_policy) and PromQL (condition_prometheus_query_language).

Usage Attribution

For all gcloud CLI, REST API, and MCP attribution requirements when executing commands with this skill, follow the instructions in Usage Attribution.

Enforce Core Alerting Rules

1. Apply the Four Golden Signals: Cover latency, traffic, errors, and saturation for the target workload.
2. Use Dynamic Grouping: Aggregate metrics by resource labels and parameterize thresholds with Terraform variables. Never hardcode ephemeral pod or instance identifiers.
3. Default to PromQL: Use Prometheus Query Language (PromQL) for metric conditions, and use log-based alerts (condition_matched_log or google_logging_metric) for lifecycle and audit events.
4. Attach Skill Attribution Labels: Always include user_labels with created-with-google-skill = "cloud-monitoring-alert-configuration" on every generated google_monitoring_alert_policy resource.
5. Require User Confirmation Before Mutating Resources: Never execute commands that modify infrastructure, enable telemetry, create online monitors, or apply Terraform without waiting for explicit user confirmation.
6. Prompt for Notification Channels: Alerting policies without notification channels incur charges without notifying anyone during an incident. Whenever notification channels are not specified in the user prompt across any workflow, explicitly inform the user that no notification channels are attached to the alerting policies, advise them to configure and attach notification channels before deploying, and ask at the end of your response if they want to configure notification channels.

Route by Target Resource Type

Identify the target resource type and read only the matching reference file directly. Never run directory listings or workspace file searches to look for queries or .tf files unless the prompt explicitly provides a file path. If a user asks to fix or validate a PromQL query without providing the query text or a file path, construct a representative broken-and-corrected query example directly.

Configure Cloud Run Alerts

• HTTP Services: cloud_run_services.md
• Batch Jobs: cloud_run_jobs.md
• Worker Pools: cloud_run_worker_pools.md

Configure Google Kubernetes Engine Alerts

• PromQL Queries and Golden Signals: GKE PromQL Queries
• Metrics Catalog and Open-Source Alerts: GKE Metrics
• Prerequisites and Setup: GKE Setup

Configure AI Agent Alerts

• Reliability: Reliability
• Quality: Quality
• Cost: Cost
• Safety: Safety
• Security: Security
• Telemetry Enablement: Telemetry
• Traffic Profiles: Historical and No History

Configure Compute Engine and AI Hypercomputer Alerts

• Compute Engine VMs, MIGs, GPU XID Insights, Host or Reservation Maintenance (type=scheduled, unscheduled, or multiple), and Machine or Domain Health: compute_engine_alerts.md

Configure Generic Resource Alerts

• Generic Workloads, Custom Metrics, MQL Migration, and HCL Deduplication: generic_alert_policies.md

Delegate Service Level Objective Alerts

Delegate to the google-cloud-slo-alert-configuration skill. That skill owns the SLO setup wizard, the burn-rate PromQL templates (references/promql_templates.md), the SLI metric mapping (references/service_metrics.md), and the App Hub scoping labels (references/app_hub_labels.md). Do NOT duplicate those templates here or claim that this skill owns its own SLO PromQL template library.

Delegate whenever the user asks for an availability or latency SLO, an error budget or burn-rate alert, or a Service Level Objective on any resource. When SLOs are only one part of a broader request (for example, a full production alerting suite for a Cloud Run service), delegate the SLO policies and continue generating the remaining golden-signal policies from the resource-specific reference in the same response.

Every resource reference carries a Context to hand off list naming the SLI metric, its good or bad filter, and the grouping labels — under a ## Delegate Service Level Objective Alerts heading for Cloud Run and generic resources, and inside the SLO section itself for GKE and Agent Platform. Read it and pass that context on, so the SLO skill does not repeat discovery you have already done. When delegating, you MUST also:

1. State the SLO target and window explicitly if the user gave them, or let the SLO skill apply its SRE defaults (99.9% over a rolling 28 days) if they did not.
2. Require the returned policies to carry user_labels with created-with-google-skill = "cloud-monitoring-alert-configuration" so attribution stays consistent with the rest of the generated suite. This literal value does NOT change when you delegate: the label records the entry-point skill, not whichever skill supplied the template. Do NOT set created-with-google-skill = "google-cloud-slo-alert-configuration" on policies generated through this delegation.
3. Preserve the burn-rate duration contract, which google-cloud-slo-alert-configuration does not specify: Fast Burn (1h and 5m windows, factor 14.4) carries duration = "300s", and Slow Burn ([3d] and [6h] lookback, factor 1.0) omits the duration field entirely — do not emit duration = "0s" in its place. Cloud Monitoring rejects a non-zero retest buffer when the lookback window exceeds 25 hours.
4. When no notification channels are provided, inform the user that no notification channels are attached to the alerting policies, advise them to configure and attach notification channels, and ask if they want to configure notification channels.

[!IMPORTANT] Always tell the user about the referral, even on the fallback path. Some reference files in this skill include standalone burn-rate templates to use when the google-cloud-slo-alert-configuration skill is not installed. If you emit policies from those fallback templates, your response MUST still tell the user that google-cloud-slo-alert-configuration owns SLO configuration, state that the emitted policies use the local fallback templates, and list the handoff context. Never silently present SLO policies as if this skill owned them.

[!NOTE] Exception: Agent Platform (OpenTelemetry-instrumented AI agents on any runtime, including Vertex AI Agent Engine and Cloud Run) is the one resource whose Fast and Slow burn policies live in this skill and remain authoritative. Delegate only for SLO customization — see Reliability.

Delegate Related Observability Tasks

• Notification Channels: Delegate to cloud-monitoring-notification-channels-configuration to configure email, PagerDuty, Pub/Sub, or Slack channels. When an alerting policy has no notification channels attached, inform the user that no notification channels are attached and that they should configure notification channels and attach them to the alerting policy.
• PromQL Authoring: Delegate to cloud-monitoring-promql-query for standalone PromQL query authoring or metric exploration.

Follow the Alert Configuration Workflow

1. Discover: Identify target resources, metric schemas, and scoping projects using the lookup commands in the matching reference file (such as discover_metric_schema.py for Compute Engine metrics and other Google Cloud resource metrics, or gather_agent_info.py, check_telemetry.py, analyze_traffic.py, create_online_monitor.py, list_trace_scope_table_names.py, and list_log_scope_table_names.py for AI agents).
2. Route: Load the reference file for the target resource type or delegate SLOs to google-cloud-slo-alert-configuration.
3. Generate: Produce parameterized Terraform HCL configurations with required user_labels.
4. Validate: When the loaded reference file requires a validation script (such as lint_promql.py, validate_config.py, lint_syntax.py, or scan_duplicates.py), run or print that exact script command. Do not list or search the scripts/ directory.
5. Explain and Prompt: Explain each alert policy in plain English. When no notification channels are provided, tell the user that no notification channels are attached to the alerting policy, advise them to configure and attach notification channels, and ask if they want to configure notification channels.