pipeline-monitor

Pipeline Monitor

Interactive skill that watches the workbuddy pipeline, diagnoses common failure modes, and applies safe fixes. Works for both the v0.5+ default bundle topology (supervisor + coordinator + worker units) and the legacy single-process serve mode.

When to use

• User asks to monitor / watch / check the pipeline
• User suspects issues are stuck or agents are failing
• User wants a post-mortem on a completed batch
• Background monitoring notified you that an agent completed or failed

Step 0: Detect deployment topology

ps -eo pid,cmd | grep -E "workbuddy (supervisor|coordinator|worker|serve)" | grep -v grep

• Bundle (default since v0.5): three processes — workbuddy supervisor
  • workbuddy coordinator + workbuddy worker. The supervisor owns the agent subprocesses behind a unix socket, so the worker can be killed and restarted without losing in-flight Claude/Codex runs.
• Serve: a single workbuddy serve process. Killing it kills agents.

For systemd installs, also run:

systemctl --user status workbuddy-supervisor workbuddy-coordinator workbuddy-worker 2>/dev/null \
  || systemctl --user status workbuddy

The CLI surface is identical across both topologies — only the failure recovery details differ (see "Bundle vs serve recovery" below).

What to check (every pass)

1. Issue state via CLI — what workbuddy thinks is happening

   workbuddy status --repo Owner/Repo
   workbuddy status --stuck                 # issues idle >1h, or cycle counts > 0
   workbuddy status --tasks --status running
   workbuddy status --events --since 10m    # audit trail
   

2. Automated diagnosis — surfaces common patterns with suggested fixes

   workbuddy diagnose                      # scan
   workbuddy diagnose --fix                # apply only safe fixes (cache invalidation etc.)
   workbuddy diagnose --repo Owner/Repo --format json   # for piping
   

3. GitHub labels — the ground truth, in case the local DB is stale

   gh issue view N -R Owner/Repo --json labels,state,comments \
     --jq '{labels: [.labels[].name], state: .state, comments: (.comments | length)}'
   

4. Agent processes — what's actually running on the host

   ps -eo pid,etime,pcpu,cmd | grep -E "codex.*exec|claude.*exec" | grep -v grep
   

5. Hooks — operator-owned event hooks (cap-hit alerts, Slack pings, etc.)

   workbuddy hooks list
   workbuddy hooks status --coordinator http://127.0.0.1:8081 \
     --token-file /home/$USER/.config/workbuddy/auth-token
   

6. Operator incident inbox — coordinator drops files here when something needs human/Claude attention (cap-hit, lost worker, dispatch refusal)

   ls -lt ~/.workbuddy/operator/inbox/ 2>/dev/null
   # operator-watch tails this and spawns Claude per incident
   

Block until something happens

Prefer status --watch over hand-rolled until loops:

workbuddy status --watch --repo Owner/Repo --issue N --timeout 30m

For a free-form wait on labels:

until gh issue view N -R Owner/Repo --json labels --jq '[.labels[].name]' \
      | grep -q "status:done"; do sleep 30; done
echo "ISSUE_COMPLETED"

Direct DB peeks (when CLI isn't enough)

Useful columns added since v0.4: claim_token, acked_at, heartbeat_at, rollout_index, rollouts_total, rollout_group_id.

sqlite3 .workbuddy/workbuddy.db <<'SQL'
SELECT agent_name, status, worker_id, heartbeat_at, lease_expires_at, created_at
  FROM task_queue
 WHERE repo='Owner/Repo'
 ORDER BY created_at DESC LIMIT 10;
SQL

Useful event types: state_entry, dispatch, completed, worker_registered, dispatch_blocked_by_dependency, claim_expired, poll_cycle_done, token_usage.

Since v0.6.24 the audit trail also carries the self-healing signals described below: dispatch_skipped_inflight (with source of statemachine, router, or resync), dispatch_skipped_agent_not_found, transition_skipped, task_reaped, periodic_recovery_tick, issue_resynced, and worker_tunnel_down findings from diagnose.

Self-healing signals (since v0.6.24)

The coordinator runs three background loops that recover the pipeline without operator intervention. Recognising their event signatures saves chasing ghosts.

• 5-minute snapshot resync (internal/poller): for every cached open issue carrying a workflow trigger label, the poller re-emits an EventIssueResynced state-entry. The state machine consults IsInflight + HasAnyActiveTask; if either gates, you see dispatch_skipped_inflight with source="resync" and trigger="issue_resynced". An issue_resynced audit row appears only when both guards pass and the issue actually gets re-dispatched. Healthy steady-state: many dispatch_skipped_inflight (one per labeled issue per 5min), zero issue_resynced.
• 60-second task_queue reaper (internal/app/task_reaper): any row with status='running' and a heartbeat older than 5 minutes (or NULL heartbeat older than 5 min from created_at) is flipped to failed, exit_code=-2, and a task_reaped event with {task_id, repo, issue_num, agent, worker_id, last_heartbeat_at, reason:"no_heartbeat"} is emitted. Watch this when a worker host dies or OOM-killer reaps a supervisor.
• 60-second periodic recovery sweep (PollerManager.recoverAllPeriodically): walks every active repo, calls recoverOrphanedActiveStates, and emits periodic_recovery_tick with {repos_swept, issues_redispatched}. After the reaper unblocks HasAnyActiveTask, this re-issues EventIssueCreated so the issue gets re-dispatched.

These three together close the silent-stall pattern from issue #345 (coordinator restart with cache already in sync → no events → nothing dispatches). If you see an issue stuck developing with the worker alive, first check whether the recovery loop is firing — a zero-count periodic_recovery_tick every 60 seconds confirms the loop is healthy and the issue truly is in-flight, not silently dropped.

Common failure modes & fixes

A. Issue not dispatched — missing workbuddy label

Symptom: issue has status:developing but no task in queue.

gh issue edit N -R Owner/Repo --add-label workbuddy

B. Task stuck running with expired lease and no agent process

Symptom: task_queue.status=running, lease_expires_at in the past, no codex/claude process for it.

Since v0.6.24 the task_queue reaper handles this automatically — within ~60s of the heartbeat aging past 5min, the row flips to failed/exit_code=-2 and a task_reaped event fires. The next periodic_recovery_tick (also ~60s) re-issues EventIssueCreated and the state machine re-dispatches. Verify via:

workbuddy status --events --since 5m --type task_reaped
workbuddy status --events --since 5m --type periodic_recovery_tick

In bundle mode this is otherwise rare — the supervisor reconciles surviving agents back to the worker on reattach. If the reaper hasn't fired yet and you don't want to wait, force it manually:

workbuddy issue restart --repo Owner/Repo --issue N \
  --coordinator http://127.0.0.1:8081 \
  --token-file /home/$USER/.config/workbuddy/auth-token
workbuddy cache invalidate --repo Owner/Repo --issue N

In serve mode (no supervisor), the reaper still runs as part of the coordinator loop. Use workbuddy recover --kill-zombies --prune-worktrees only when you need to clean up host-side worktrees / pids; hand-editing task_queue rows remains discouraged (claim tokens / heartbeats interact in non-obvious ways).

C. Worker not picking up a pending task

Symptom: a pending task exists, but the worker is idle. Often an older running row with a stale lease blocks the claim query.

workbuddy diagnose --fix          # reaps stale claims, invalidates cache
# If still stuck after the next poll cycle:
workbuddy issue restart --repo Owner/Repo --issue N

D. Dev-agent self-blocks — missing Acceptance Criteria

Symptom: dev-agent completes in ~1 min and sets status:blocked. Edit the issue body to add a ## Acceptance Criteria section, then:

gh issue edit N -R Owner/Repo --remove-label 'status:blocked' --add-label 'status:developing'
workbuddy cache invalidate --repo Owner/Repo --issue N

E. Agent completes but doesn't change labels

Symptom: task completed, agent process gone, but labels unchanged. The LLM didn't run the routing gh issue edit. The coordinator self-heals this on the next poll, but you can shortcut:

gh issue edit N -R Owner/Repo --remove-label 'status:developing' --add-label 'status:reviewing'

F. Review sends issue back to developing (normal retry)

Label flips reviewing → developing. Do not intervene. The 3-cycle cap (REQ-055) auto-stops dispatch and posts a needs-human comment.

G. Coordinator not detecting label changes

Symptom: labels changed on GitHub but no state_entry event.

workbuddy cache invalidate --repo Owner/Repo --issue N

H. Codex stuck in API inference (most common runtime failure)

Symptom: codex process alive at ~0% CPU, no children, no JSONL writes for 10+ min. Often after the agent has already changed labels.

# Find the live session log
ls -lt .workbuddy/sessions/ | head
# Confirm staleness
for f in .workbuddy/sessions/*/codex-exec.jsonl; do
  age=$(( $(date +%s) - $(stat -c '%Y' "$f") ))
  [ $age -gt 600 ] && echo "STALE ($age s): $f"
done
# Confirm no child processes
pstree -p <codex-pid>

Fix: kill the codex pid. In bundle mode the worker reattaches and records a clean exit; in serve mode you may need workbuddy recover --kill-zombies to clean up. Then check labels — if they already advanced, just cache invalidate and let the next poll move on; otherwise workbuddy issue restart.

I. Issue claim stuck after coordinator crash (REQ-057/059)

Symptom: no dispatch even after cache invalidate; diagnose shows a claim still held by a dead coordinator/worker.

• Self-heal: claims have a TTL — wait for expiry, the next poll emits claim_expired and overwrites.
• Self-heal (v0.6.24+): the periodic-recovery sweep + reaper combo picks up stale running rows within ~2 minutes total. Watch for task_reaped followed by periodic_recovery_tick with issues_redispatched >= 1 and a fresh dispatch event for the issue.
• Force: workbuddy recover clears local runtime state; for a remote coordinator's in-memory inflight tracker, use workbuddy issue restart --coordinator <url>.

N. Coordinator restart with cache already in sync (silent-stall, REQ-150/151/152, fixes #345)

Symptom: coordinator was restarted; eligible issues sit at their existing status:reviewing / status:developing labels with no task_enqueued, no dispatch, no error — only poll_cycle_done events appear.

This is the pre-v0.6.24 silent-stall. The poller diffs labels against its cache; when the cache is already in sync after restart, no event fires and the state machine is never invoked.

Self-heal (no manual action needed since v0.6.24):

• Within 60s the periodic-recovery sweep enumerates cached open issues carrying a workflow trigger label and re-emits EventIssueCreated via recoverOrphanedActiveStates.
• Within 5min the snapshot resync emits EventIssueResynced on every poll cycle, redundant defense in case the recovery sweep is gated by a stale running task row (which the reaper handles, see fix B).
• Confirm by watching:

  workbuddy status --events --since 10m \
    --type periodic_recovery_tick \
    --type issue_resynced \
    --type dispatch_skipped_inflight
  

  Healthy cadence: 10 periodic_recovery_tick events + 2 dispatch_skipped_inflight (source=resync) per labeled issue in 10 minutes. Zero issue_resynced rows is also healthy — that audit fires only when a resync actually re-dispatches.

J. Consecutive-failure cap reached (REQ-055)

Symptom: diagnose reports "dev-agent has failed 3 times in a row", dispatch stops, issue gets a needs-human comment.

• Read the latest comments: "Infra Error" header → launcher/runtime crash (REQ-056), doesn't burn the budget; fix infra, then restart. A plain "Failure" header → agent disagrees with the AC; tighten AC or intervene.
• Reset: fix the root cause, flip label back to status:developing, workbuddy cache invalidate --repo Owner/Repo --issue N.

K. Worktree setup failed — worker refuses to fall back to CWD (REQ-058)

Symptom: issue comment "worktree setup failed"; task marked failed.

workbuddy recover --prune-worktrees --dry-run    # preview
workbuddy recover --prune-worktrees              # apply
workbuddy issue restart --repo Owner/Repo --issue N

M. Issue not dispatching because a dependency is blocking it

Symptom: issue is status:developing with the workbuddy label, no agent process, no dispatch event in workbuddy status --events. The issue may also carry a ? reaction added by workbuddy.

The Coordinator gates dispatch on workbuddy.depends_on declared in the issue body (YAML block under ## Dependencies). When verdict is blocked or needs_human, dispatch is refused and a dispatch_blocked_by_dependency event is logged.

Diagnose:

workbuddy status --repo Owner/Repo            # CYCLES + DEPENDENCY columns
workbuddy status --events --type dispatch_blocked_by_dependency --since 1h
workbuddy diagnose --repo Owner/Repo          # surfaces malformed/cyclic deps
gh issue view N -R Owner/Repo --json body --jq .body | grep -A5 'workbuddy:'

Fix by intent:

• Blocker really not done yet → wait, or work the upstream issue.
• Blocker actually done, declaration stale → edit the issue body to drop the satisfied depends_on ref, then workbuddy cache invalidate --repo Owner/Repo --issue N.
• Need to override a real dependency for one run → add the override:force-unblock label (verdict flips to override). Don't hand-toggle status:blocked — the Coordinator owns that label.
• Malformed depends_on ref → diagnose points at the offending line; fix the YAML, cache invalidate.
• Cross-repo ref (owner/other#N) — accepted-but-unsupported in v0.1; verdict goes to needs_human until the upstream lands or the ref is removed.

L. Worker restart didn't kill agents — that's by design (bundle only)

Bundle mode preserves agent runs across worker restarts via the supervisor socket. If you systemctl --user restart workbuddy-worker and ps still shows codex/claude — that's correct, not a leak. The new worker reattaches and resumes the JSONL stream.

Bundle vs serve recovery cheat sheet

Symptom	Bundle (default)	Serve
Worker crashed mid-task	Supervisor keeps the agent; worker reattaches on restart	Agent dies with the process; use recover --kill-zombies
Need to upgrade workbuddy without dropping runs	workbuddy deploy upgrade --name workbuddy-worker	Drain or accept downtime
Local runtime cleanup	workbuddy recover --prune-worktrees (don't --kill-zombies lightly — supervisor is managing them)	workbuddy recover --kill-zombies --prune-worktrees

Decision tree

Detect topology (bundle vs serve)
│
├─ workbuddy diagnose --repo … (start here, every time)
│
├─ Check GitHub labels for target issues
│  ├─ All status:done → Report. Stop.
│  ├─ status:developing/reviewing + active agent process → Normal. Monitor.
│  └─ Label unchanged for >10 min with no agent process → see fixes B/E/H
│
├─ workbuddy status --tasks --status running / --stuck
│  ├─ running + agent alive → Normal. Wait or `status --watch`.
│  ├─ running + no agent → fix B
│  ├─ pending + worker alive → fix C
│  └─ no pending + labels unchanged → fix A or G
│
└─ Logs / journal
   ├─ journalctl --user -u workbuddy-coordinator -n 100   # bundle
   ├─ journalctl --user -u workbuddy-worker      -n 100   # bundle
   └─ journalctl --user -u workbuddy             -n 100   # legacy serve

Reporting template

## Pipeline Status Report

| Issue | Repo | State | Cycles | Agent | Notes |
|-------|------|-------|--------|-------|-------|
| #37 | AegisLab | reviewing | 2/3 | review-agent | In progress |
| #88 | workbuddy | developing | 0/3 | dev-agent | Started |

**Topology:** bundle (supervisor + coordinator@127.0.0.1:8081 + 1 worker)
**Repos:** Lincyaw/workbuddy, OperationsPAI/AegisLab
**Live agents:** 1 codex
**Interventions:** `issue restart` for #37 after stale claim

Arguments

Optional: <owner/repo> <issue-numbers...> Example: /pipeline-monitor OperationsPAI/AegisLab 37 42 If omitted, monitor all repos and all open workbuddy-labeled issues.