pilot-watch-alert-response

pilot-watch-alert-response

The real-time event-driven scenario from Pilot's side. Pilot subscribes to cub-scout watch or cub-scout bot (which expose the same webhook / file event sinks) and, for each event, decides PASS / WATCH / ASK / BLOCK — calling back into cub-scout for context as needed. This is the only event-driven skill in the pilot- batch*; the other four are query-driven.

With the receipts v2 surface shipped (#463), Pilot can ask cub-scout watch or cub-scout bot to inline a receipt with each matching event via --emit-receipt-on drift.detected,ownership.changed or --emit-receipt-on all. That delivers a fingerprinted in-toto Statement v1 per event in the same JSONL line — Pilot reads the verdict + receipt in one shot, no synchronous call-back required for the common case.

When to use

Explicit phrasings:

• "Pilot subscribes to the watch stream"
• "Render verdicts on each drift event as they fire"
• "Real-time acceptance kernel for cluster changes"
• "Pilot, decide on this drift.detected event"
• "What does Pilot say about the ownership.changed alert"
• "Stream PASS/WATCH/ASK/BLOCK per event from cub-scout watch"
• "Run the same stream from cub-scout bot in the cluster"
• "Pilot reads receipts inline from the watch event stream"
• "Trigger Pilot on any cub-scout drift event"

Implicit intents:

• The flow is event-driven, not query-driven (Pilot reacts to a stream, not a polling job)
• The decision needs to fire fast — a Pilot verdict per event, ideally inline
• The receipt-attached path (v2 --emit-receipt-on) is preferred when audit durability matters
• Backpressure is a real concern when events fire fast (e.g., ApplicationSet expansion)

Do not load for

• A single-resource pre-deploy gate — pilot-cd-gate
• A periodic fleet roll-up verdict — pilot-fleet-conformance
• A drift-classification deep-dive for one resource — pilot-patch-and-drift
• A postmortem evidence pack — pilot-incident-evidence
• Pure observation without a Pilot policy decision — scout-observe covers the watch surface from the operator side
• Any mutating action. Pilot's mutation path is cub / Argo / Flux / kubectl — never cub-scout.

The Pilot ↔ cub-scout watch / bot loop

   cub-scout watch -n prod \
     --webhook https://pilot.acme/cub-scout-events \
     --emit-receipt-on drift.detected,ownership.changed \
     --output-file /var/log/cub-scout-watch.jsonl
        │
        │  ▼ JSONL event stream (one event per line)
        │  {
        │    "type": "drift.detected",
        │    "timestamp": "2026-05-22T11:00:00Z",
        │    "resource": {"kind": "Deployment", "name": "api", "namespace": "prod"},
        │    "owner": {"type": "argo", "name": "payments-api"},
        │    "severity": "warning",
        │    "details": {"category": "DRIFT"},
        │    "receipt": { in-toto Statement v1 with applied-matches-spec verdict }
        │  }
        ▼
   Pilot's event handler
        │  fast path: the event already carries a receipt
        │    │
        │    └► Pilot reads receipt.predicate.verdict → render verdict
        │
        │  slow path: the event doesn't carry a receipt
        │    │  (event type isn't in --emit-receipt-on, OR receipt-build failed)
        │    │
        │    ├──► cub-scout compare three-way <r> -n <ns> --format json
        │    ├──► cub-scout explain <r> -n <ns> --presentation ai
        │    └──► cub-scout receipt verify <r> -n <ns> --strategy <s> --out event-receipt.json
        │
        ▼
   Pilot's per-event verdict
        │  PASS  → log + continue
        │  WATCH → alert + log + (optional) attach receipt to a ticket
        │  ASK   → page on-call; the proof gap is structured in the receipt
        │  BLOCK → halt promotion / trigger Pilot's escalation policy
        └────────────────────────────────────────────────────

Event types (cub-scout watch / cub-scout bot + receipts v2)

cub-scout watch and cub-scout bot emit four event types today; --emit-receipt-on can attach receipts to any of them, with --emit-receipt-batch-cap bounding first-poll bursts:

Event type	When it fires	Receipt-built in v1?	Pilot's typical reaction
resource.discovered	First poll sees a resource cub-scout hadn't seen before	Yes — bounded by --emit-receipt-batch-cap to avoid initial burst floods.	Log; no verdict needed.
ownership.changed	A resource's ownership controller changed (Argo → Flux, Flux → kubectl, etc.)	Yes — applied-matches-spec auto-detected. Receipt verdict reflects whether the new owner has a resolved git anchor.	WATCH (often a transient state during migration); ASK if applied-matches-spec returns INCONCLUSIVE.
drift.detected	compare three-way finds a divergence on a resource cub-scout watched into	Yes — applied-matches-spec auto-detected. Receipt verdict reflects whether the drift is controller-drift (PASS / reconciling) or manual-edit (BLOCK).	Drives pilot-patch-and-drift shape: PASS = ACCEPT; WATCH = INVESTIGATE; BLOCK = REVERT/QUARANTINE.
scan.finding	A risk pattern matched on a resource (CVE, expiring cert, etc.)	Yes — bounded by --emit-receipt-batch-cap to avoid initial scan floods.	Severity-driven; route to security/SRE escalation policy.

The --emit-receipt-on all sugar accepts every known event type for forward-compat. Receipt-build failures are non-fatal — the underlying event still emits, just without the receipt key (uses omitempty; consumers should check key presence, not null-ness).

# Idiomatic Python consumer
for line in stream:
    event = json.loads(line)
    if "receipt" in event:        # key-present check, NOT `is not None`
        verdict = pilot.decide_from_receipt(event["receipt"])
    else:
        verdict = pilot.fallback_decide(event)  # slow path
    handle(verdict)

Wire shape

A drift.detected event with an attached receipt (one JSONL line, pretty-printed here):

{
  "type": "drift.detected",
  "timestamp": "2026-05-22T11:00:00Z",
  "resource": {
    "kind": "Deployment",
    "name": "payments-api",
    "namespace": "prod"
  },
  "owner": {
    "type": "argo",
    "name": "payments-api"
  },
  "severity": "warning",
  "details": {
    "category": "DRIFT"
  },
  "receipt": {
    "_type": "https://in-toto.io/Statement/v1",
    "subject": [
      {
        "name": "k8s-live://apps/v1/Deployment/prod/payments-api",
        "digest": {"sha256": "abc..."}
      }
    ],
    "predicateType": "https://cub-scout.dev/receipt/v1",
    "predicate": {
      "version": "v1",
      "predicateName": "applied-matches-spec",
      "verdict": "BLOCK",
      "claim": "applied matches spec at apps/prod/payments-api",
      "scope": {"kind": "Deployment", "name": "payments-api", "namespace": "prod"},
      "evidence": {
        "attribution": {"cause": "manual-edit", "managerHint": "kubectl-edit"},
        "gitSource": {"repoUrl": "https://github.com/org/payments", "revision": "abc123", "path": "apps/prod/payments-api"}
      },
      "omissions": [],
      "inputAttestations": [],
      "nextSteps": [
        {
          "actionType": "read-only",
          "reason": "live state was last written by a kubectl-* tool; the GitOps loop was bypassed since the controller's last apply",
          "nextCommand": "kubectl get --show-managed-fields",
          "nextSurface": "kubectl"
        }
      ],
      "verifier": {"tool": "cub-scout", "version": "v2.2.1"},
      "verifiedAt": "2026-05-22T11:00:00Z",
      "fingerprint": "sha256:1a2b3c..."
    }
  }
}

Notes on the wire shape (verified against pkg/agent/receipt.go):

• predicate.version is the literal string "v1" (PredicateVersion), not "1".
• The nextSteps array is populated for the manual-edit BLOCK branch — EvaluateAppliedMatchesSpec appends one read-only next step pointing at kubectl get --show-managed-fields. (nextSteps uses omitempty so other branches that produce zero next steps omit the key entirely.)
• omissions: [] and inputAttestations: [] are explicit arrays (never null). The full receipt's fingerprint covers both.

Pilot reads receipt.predicate.verdict directly: BLOCK → halt; WATCH → alert; PASS → log + continue; INCONCLUSIVE → ASK. Receipt verdicts are restricted to those four — ASK is a source-truth status value (and maps to receipt WATCH when wrapped, per pkg/agent/receipt_predicates.go:396-407), not a receipt verdict.

Step-by-step

Step 1 — confirm the watch stream is shaped for Pilot

# Pre-flight: doctor + status
$ cub-scout doctor
$ cub-scout status

# Confirm the event taxonomy Pilot expects
$ cub-scout watch --help | grep -A 3 '\-\-emit-receipt-on'

If standalone (no ConfigHub auth), the receipts in the stream omit the confighub-unit:// subject and record an OmissionConfigHubUnitSubject entry — structural honesty about what's missing, not a verdict downgrade. The verdict follows the predicate's normal rules; applied-matches-spec can still PASS in standalone mode. Pilot's policy may choose to treat standalone receipts more conservatively, but that's Pilot-side; cub-scout doesn't impose a ceiling.

Step 2 — start the watch stream or bot

$ cub-scout watch -n prod \
    --webhook https://pilot.acme/cub-scout-events \
    --emit-receipt-on all \
    --output-file /var/log/cub-scout-watch.jsonl \
    --severity warning,critical

• --webhook is Pilot's listener (POST per event).
• --output-file keeps a local JSONL log (useful for replay during postmortems; see pilot-incident-evidence).
• --emit-receipt-on attaches receipts to matching event types; use all with the default batch cap when first-poll bursts are possible.
• --severity filters scan-finding events to the actionable subset.

The watch loop is read-only by design. cub-scout's only cluster operations are Get / List / Watch; --emit-receipt-on adds an in-process BuildReceipt per matching event, which is also pure (the receipts code path is statically guarded by TestReceiptPackageReadOnlyClient).

Step 3 — Pilot's per-event handler

Pilot's logic, per event:

def handle_event(event: dict) -> Verdict:
    # Fast path: event carries an inline receipt
    if "receipt" in event:
        receipt = event["receipt"]
        verdict_str = receipt["predicate"]["verdict"]  # PASS|WATCH|BLOCK|INCONCLUSIVE
        # The receipt also carries omissions[], attribution, gitSource — Pilot can drill in
        if verdict_str == "PASS":
            return Verdict.PASS
        elif verdict_str == "WATCH":
            attach_receipt_to_ticket(receipt)
            return Verdict.WATCH
        elif verdict_str == "BLOCK":
            escalate(event, receipt)
            return Verdict.BLOCK
        elif verdict_str == "INCONCLUSIVE":
            return Verdict.ASK

    # Slow path: no inline receipt (event type not in --emit-receipt-on, OR receipt-build failed)
    return synchronous_evidence_gathering(event)

The slow path calls back into cub-scout synchronously:

$ cub-scout compare three-way deploy/payments-api -n prod --format json > evidence.json
$ cub-scout receipt verify deploy/payments-api -n prod --save --out event.receipt.json

Step 4 — verify the receipt fingerprint (optional but recommended)

Pilot may want to verify that the inline receipt's fingerprint is intact — particularly when the event arrived via a webhook that traversed network boundaries. cub-scout's receipt validate path is exit-coded:

$ echo "$event" | jq '.receipt' > event-receipt.json
$ cub-scout receipt validate event-receipt.json
# exit 0 = fingerprint matches; 1 = mismatch (refuse to act on it); 2 = I/O

Operational defaults Pilot should know

• Receipt-build failures are non-fatal: the event still emits, the receipt key is omitted (omitempty), and cub-scout logs a stderr warning. Pilot's handler must check for key presence, not null-ness.
• Warnings are rate-limited (Codex round-6 P2 fix in #463): the first 10 receipt-build warnings pass through verbatim; afterwards, a summary line fires every 100 suppressed warnings. A long-running watch with transient cluster-read failures won't spam stderr.
• Unsupported event types pass through silently: resource.discovered and scan.finding accept the flag (so --emit-receipt-on all works) but cub-scout skips receipt-build for them in v1. A startup warning lists them.
• Backpressure is not yet handled (open in #449): if events fire faster than the consumer drains, the watch can queue up to --max-queued-events and then drops oldest. ApplicationSet expansion is the canonical "events fire fast" scenario.
• Connected vs standalone: receipts attached to events include the confighub-unit:// subject only when connected. Standalone-mode receipts omit it and carry an OmissionConfigHubUnitSubject entry — that's structural honesty about missing evidence. The verdict itself follows the predicate's normal rules (PASS is possible in standalone mode). Pilot may apply its own conservative policy; cub-scout doesn't impose a ceiling.

Worked example: drift event triggers Pilot

A drift.detected event fires for deploy/payments-api in prod. Pilot's webhook receives:

{
  "type": "drift.detected",
  "timestamp": "2026-05-22T11:00:00Z",
  "resource": {"kind": "Deployment", "name": "payments-api", "namespace": "prod"},
  "owner": {"type": "argo", "name": "payments-api"},
  "severity": "warning",
  "details": {"category": "DRIFT"},
  "receipt": {
    "_type": "https://in-toto.io/Statement/v1",
    "predicateType": "https://cub-scout.dev/receipt/v1",
    "predicate": {
      "predicateName": "applied-matches-spec",
      "verdict": "BLOCK",
      "evidence": {
        "attribution": {"cause": "manual-edit", "managerHint": "kubectl-edit"}
      },
      "fingerprint": "sha256:1a2b3c..."
    }
  }
}

Pilot's handler:

1. Sees "receipt" in event → fast path.
2. Reads receipt.predicate.verdict: "BLOCK".
3. Reads receipt.predicate.evidence.attribution: {cause: manual-edit, managerHint: kubectl-edit} → confirms human kubectl-edited an Argo-owned resource.
4. Renders Pilot verdict: BLOCK.
5. Escalates: opens a ticket, attaches the receipt, pages the on-call.

Total round-trip: one webhook POST, zero synchronous cub-scout calls. The receipt is fingerprint-verifiable later if the ticket is audited.

If the same event had arrived without an inline receipt (e.g., --emit-receipt-on wasn't set), Pilot would call cub-scout compare three-way deploy/payments-api -n prod --format json to get the same evidence on the slow path.

Open scope on the cub-scout side (issue #449)

This skill calls out gaps that may surface during real deployment. They live in the open scope of #449, not this skill:

• More event types: attribution.cause-flipped (a resource's cause changes from controller-drift → manual-edit); source-truth.verdict-changed (a compare source-truth verdict flips PASS → WATCH); bindingSource.changed (a Link now resolves to a different upstream). All v2+ scope.
• Backpressure / batching: high-frequency events from ApplicationSet expansion need a documented drop / batch policy. Currently --max-queued-events controls in-process queue depth; downstream filtering is Pilot's call.
• Pilot-shaped emit format: a --format pilot or --shape acceptance-event that bundles enough context per event for Pilot to verdict-render without any synchronous call-back. Receipts inline already cover the common case, but a fuller shape could include compareThreeWay per event.
• resource.discovered / scan.finding receipt support: gated on the backpressure design (would flood on initial poll burst). Open in #449.

Filing these is a non-goal of this skill; surfacing them during real-deployment authoring is a goal.

Tool boundary

• Allowed (cub-scout): watch (not auto-approved: its --webhook flag sends events to any URL, so the host asks each time), Compare verbs, explain, trace, doctor, map, receipt verify / show / validate / list. All read-only.
• Allowed (cluster): kubectl get/describe. NOT kubectl apply / edit / patch / delete.
• Allowed (ConfigHub): cub <entity> get / list for the entities named in allowed-tools. NOT cub * create / update / delete.
• Allowed (controllers): argocd app get, flux get. NOT argocd app sync, flux reconcile.
• Pilot's mutation path (REVERT, QUARANTINE, escalation) is out-of-scope for this skill.

References

• scout-observe — the watch surface from the operator side
• scout-verify — receipt-emit (the --emit-receipt-on flag wiring)
• scout-compare — Compare verbs Pilot falls back to on the slow path
• pilot-cd-gate — pre-deploy companion (query-driven)
• pilot-fleet-conformance — periodic fleet companion (query-driven)
• pilot-patch-and-drift — drill-down for BLOCK / WATCH events
• pilot-incident-evidence — replay watch logs during postmortems
• Wire format: docs/reference/json-contracts.md § Receipt Contract — Watch event emission (v2.2)
• Receipts v1+v2: #446 (v1), #449 (this flag), #463 (the shipping merge)
• Read-only triad: scout-observe, scout-attribute, scout-compare, scout-verify

Constraints

• As of v2.3.0 (PR #470), all four known event types build receipts: drift.detected, ownership.changed, resource.discovered, scan.finding. The per-poll cap --emit-receipt-batch-cap N (default 10) bounds first-poll bursts that would otherwise flood — the first N events get receipts and the rest emit with the receipt key omitted plus a single stderr summary line.
• The receipt key uses omitempty — consumers MUST check key presence, not null-ness. event["receipt"] raising KeyError is the correct empty signal; if event["receipt"] is None is the wrong pattern.
• The wire format is the in-toto Statement v1 envelope wrapping https://cub-scout.dev/receipt/v1. Pilot's receipt parser should be schema-aware to handle future envelope additions (e.g., signing layer).
• --emit-receipt-on requires the v2.2+ cub-scout binary (the flag shipped in #463); the per-poll cap requires v2.3+ (#470). Older binaries don't have the flag and the watch event payload won't include a receipt field.
• Per-poll backpressure semantics: the cap resets every call to attachReceiptsIfRequested (one watch poll = one call), so a long-running watch with quiet polls between bursts doesn't accumulate suppression state. --emit-receipt-batch-cap 0 disables receipt-build entirely while keeping the flag explicit; large values (e.g., 1000) effectively disable the cap. See docs/reference/watch-events.md for the full specification.