investigate-drift

investigate-drift

The drift-investigation loop. Live state does not match desired state, so compare the values and inspect available provenance. Composes compare three-way (DRY/WET/LIVE) + compare drift (file/bundle vs cluster) + attribution (cause / managerHint / gitSource / bindingSource). Manager evidence classifies recognized managers; it does not establish write order or human identity.

When to use

Explicit phrasings:

• "Why is deploy/api different from the git manifest?"
• "Is a recognized interactive manager recorded for this path?"
• "Which recognized controller or interactive manager is recorded?"
• "controller-drift or manual-edit?"
• "Find the field that diverged and show its available manager evidence"
• "Compare three-way + attribution for deploy/api in prod"
• "The cluster says replicas=1, ConfigHub says 3 — what happened?"

Implicit intents:

• The user knows there's a divergence; they need the per-field detail
• The user wants the cause classified, not just the diff
• The user is planning a response (revert to controller / port edit to git / accept manual change)
• The user may need to attach the result to a release-gate decision

Do not load for

• A pager-pressed triage where the workload is broken — triage-unhealthy-workload
• Pure inventory ("what's running") — scout-observe
• Generating a fingerprinted evidence receipt — scout-verify, specifically the no-manual-edits-since predicate
• Audit-fleet-wide conformance ("which clusters are outliers?") — audit-fleet-conformance
• Any mutating fix — cub-scout never applies. Route to kubectl rollout undo, cub for ConfigHub updates, or a git commit.

The loop

1. Detect the diff with compare three-way (connected) or compare drift --file <yaml> (standalone).
2. Inspect manager evidence on each compareFieldMismatch. Path-specific classifications require decodable FieldsV1 and a mapped canonical path; otherwise the result can be a resource-level fallback, which must not be attributed to that field:
   • cause: controller-drift / manual-edit / unknown
   • managerHint: a representative recognized K8s managedFields manager string for the path classification or fallback
   • gitSource: where the desired value comes from (repo / revision / path / file:line if stage B back-resolution is wired)
   • bindingSource (connected): which ConfigHub Link supplied this field
3. Interpret the manager string conservatively using references/verified-manager-strings.md:
   • argocd-controller / kustomize-controller / helm-controller → recognized controller-manager evidence; this does not show reconciliation is happening now
   • kubectl-edit / kubectl-patch / kubectl-apply → recognized interactive-manager evidence; it does not identify the person or prove write order
   • kubectl-client-side-apply + Argo owner → Argo CSA migration co-signal; manager strings alone do not identify a human
4. Decide the response based on cause + the operator's policy. cub-scout doesn't pick; the user does.

Step-by-step

Step 1 — detect the diff

Connected mode (preferred): DRY (ConfigHub intent) vs WET (rendered) vs LIVE (cluster).

$ cub-scout compare three-way deploy/api -n prod --format json
{
  "agreement": "diverged",
  "summary": { "agreed": 5, "diverged": 2 },
  "fieldMismatches": [
    {
      "path": ".spec.replicas",
      "dry": 3, "wet": 3, "live": 1,
      "attribution": {
        "cause": "manual-edit",
        "managerHint": "kubectl-edit",
        "gitSource": { "repoUrl": "https://github.com/org/platform-config", "revision": "abc123", "path": "apps/prod/api" }
      }
    }
  ]
}

Standalone mode: compare drift --file desired.yaml -n prod for a file-vs-live diff. Or compare three-way --source-path <local-git-checkout> for stage B back-resolution (file:line per field).

Step 2 — read the attribution

Each compareFieldMismatch.attribution block carries the per-field provenance. The fields:

Field	Meaning
cause	One of controller-drift / manual-edit / unknown (verified manager-string enumeration in pkg/agent/manager_strings.go)
managerHint	A representative recognized manager string; not proof of the latest writer or a person
gitSource.{repoUrl, revision, path}	Resource-level git anchor from the Argo / Flux tracer (A2)
gitSource.{file, line}	Stage B back-resolution (--source-path <local-checkout>) — the exact YAML file:line where the field was set
bindingSource (connected)	{unit, path, link} — which ConfigHub Link supplies this field's value (C2)

The cause classifies recognized manager evidence. managerHint is a manager string, not a person or timestamped writer identity. The git/binding sources point to desired-state provenance.

Step 3 — interpret manager evidence

Read the manager-string enumeration in references/verified-manager-strings.md. These are manager classifications, not proof of a current reconciliation or human action:

Writer	Owner context	Meaning
argocd-controller	Argo-owned	Recognized Argo controller-manager evidence. controller-drift; recency is not established.
kubectl-client-side-apply	Argo-owned	Argo CSA migration co-signal. controller-drift; does not establish recency.
kubectl-client-side-apply	Native (no Argo signal)	Classified as interactive-manager evidence. manual-edit; the human identity is unknown.
kustomize-controller / helm-controller	Flux-owned	Recognized Flux controller-manager evidence. controller-drift; recency is not established.
kubectl-edit / kubectl-patch	Any	Recognized interactive-manager evidence. manual-edit; person and write order are unknown.
apiextensions.crossplane.io/composed-<hash>	Crossplane-owned	Recognized Crossplane composed-resource manager evidence. controller-drift; recency is not established.
helm (bare)	Helm-direct	Helm manager evidence. controller-drift for OwnerHelm; it does not establish a person or write order.
Unrecognized	Any	cause=unknown. cub-scout parses; it does not guess.

The owner co-signal is essential for the kubectl-client-side-apply disambiguation. See observe-argocd for the full rule.

Step 4 — decide the response

cub-scout produces the evidence; the operator picks the action.

Cause	Common operator response (mutations are user-driven)
controller-drift + independently observed recent sync	Compare live values again and inspect controller health; manager evidence alone does not show it is reconciling or that the diff is transient.
controller-drift + persistent	Investigate controller status (Argo OutOfSync, Flux False, kstatus failure); do not infer a broken reconciler from manager evidence alone.
manual-edit + change should be kept	Verify the live value and desired source, then port an approved change back to git.
manual-edit + change should be reverted	Verify policy and live value, then have the operator restore desired state; do not assume which value is newest.
manual-edit + change should be accepted as canonical	Verify and review the live value, then update the source of truth through the operator's normal process.
unknown	Cannot classify. Missing, incomplete, unmapped, or unrecognized evidence can all yield unknown; investigate the available metadata without guessing.

Worked example

Production deploy/api has 1 replica; ConfigHub says 3. Argo CD is in Synced state. The operator's question: "did Argo do this, or did someone?"

$ cub-scout compare three-way deploy/api -n prod
Three-way compare for Deployment/api in prod

Agreement: DIVERGED — 2 of 7 fields disagree

Field mismatches:
  .spec.replicas
    DRY (ConfigHub unit payments-api rev=42):    3
    WET (rendered):                              3
    LIVE (cluster):                              1
    Cause:        manual-edit (manager: kubectl-edit)
    Git source:   https://github.com/org/platform-config @abc123 path=apps/prod/api
  .spec.template.spec.containers[0].resources.requests.cpu
    DRY:  500m
    WET:  500m
    LIVE: 250m
    Cause:        manual-edit (manager: kubectl-patch)
    Git source:   (same)

Reading: If these are path-specific classifications, each path has recognized interactive-manager evidence; controller-manager evidence may also be present. The strings do not prove separate commands, identify a person, or establish that Argo's last sync predates these values. Check the actual live and desired values and the controller status independently.

Response options:

• If the live values should return to desired state, have the operator restore them through the approved workflow and verify reconciliation.
• If the live values should become canonical, have the operator review and update the source of truth through the normal workflow.
• Manager evidence alone does not establish that somebody bypassed GitOps. For time-bounded history, use a suitable audit source or the receipt predicate with its documented timestamp evidence.

cub-scout produced the evidence in one command. The decision is the operator's.

Connected-mode enrichment

With cub auth login, compare three-way adds:

• DRY column — the ConfigHub Unit's intent (the canonical "what we asked for")
• bindingSource per field — which ConfigHub Link supplies this value (e.g., link:env-shared.replicas). Per-field provenance into the unit graph.
• confighubUrl — deep-link to the unit's revision in the ConfigHub GUI
• incomingBindings[] on the unit — which upstream units feed this unit

Standalone mode loses the DRY column and bindingSource; what remains is the WET (file) vs LIVE (cluster) diff. A path-specific manager classification is available only when decodable FieldsV1 maps the mismatch; otherwise output may be resource-level or unknown. It still does not establish write order or a person.

Tool boundary

• Allowed: the four Compare verbs; explain, trace; kubectl get/describe/get --show-managed-fields; cub <entity> get / list for the entities named in allowed-tools (connected); argocd app get, flux get (controller-side reads)
• Not allowed: kubectl rollout undo, kubectl edit, kubectl patch, argocd app sync, flux reconcile, cub * update/create/delete. The investigation produces evidence; the response is the operator's. cub-scout's compare three-way --fail-on and --suggest flags work for CI gating but do not write.

References

• scout-compare — the verb group this skill composes
• scout-attribute — the attribution-layer evidence surface
• references/kubernetes-managedfields.md — the data substrate
• references/verified-manager-strings.md — the writer enumeration
• observe-argocd, observe-flux — owner co-signals for disambiguation
• scout-verify — the no-manual-edits-since predicate captures the same evidence as a fingerprinted receipt
• Attribution layer issues: #435 (parent), #437 (A1+A1.5+A2), #438 (C1), #439 (C2), #440 (stage B)
• Example: examples/drift/mutation-cause-attribution/

Constraints

• This skill is investigation, not remediation. cub-scout categorically never applies a fix. Every revert / port / accept is the operator running another tool.
• cause=unknown is honest when manager evidence is missing, incomplete, unmapped, or unrecognized. Don't pressure-classify it.
• controller-drift and manual-edit describe recognized manager presence in the resource or mapped path. The classifier does not use manager timestamps to order writers or prove a person acted; inspect audit logs for identity/history.
• Stage B file:line back-resolution requires --source-path <local-git-checkout> and only covers raw YAML (no Helm / Kustomize templating yet — see #435 follow-ons).