resume-run

Resume a run

resume continues the same run (same run id, same lineage) from the last completed iteration. To instead start a new run seeded from one or more prior runs, see ../import-run/SKILL.md.

run_root layout and the checkpoint/snapshot model are described in docs/using-eve.md.

1. Pause (if the run is live)

Kill the process at any time — Ctrl-C, SIGTERM, or tmux kill-session. Do not wait for a "safe" boundary. Each completed iteration is snapshotted, so an interrupted iteration is recoverable; the cleanup happens on resume, not on kill.

2. Adjust (optional, before resuming)

These take effect on resume (the run re-reads them from disk each iteration):

• the optimizer's immutable/ and prompt/ files
• the run config

Does NOT take effect on resume: editing initial_guidance/. The optimizer population is restored from the run's lineage DB, not re-seeded. To change the optimizer itself, start a fresh run instead — see ../import-run/SKILL.md.

3. Resume

uv run python -m scaling_evolve.algorithms.eve.runner --config-name=<config> resume_from=<absolute_run_root>

The runner does the recovery for you: it reads <run_root>/checkpoint.json for the last completed iteration, rolls the solver/optimizer lineage DBs back to that iteration's snapshot, restores local CSV telemetry to that same iteration, archives interrupted post-checkpoint workspaces under <run_root>/.resume_archive/, and continues under the same run id.

• resume_from must be an absolute path; a relative path is rejected.
• If you launched without wandb, re-pass logger.wandb.enabled=false here too (logger overrides are not persisted across the resume invocation).
• Optional: resume_iteration=<K> rolls back to an earlier completed iteration K (must be <= last_completed_iteration).
• Local telemetry under <run_root>/telemetry/ is logical experiment output. After resume, it should contain rows through the checkpoint anchor plus newly completed post-resume rows, not rows from the failed partial attempt.
• .resume_archive/resume_*/manifest.json records any active post-anchor workspace paths that were moved out of the logical run surface. Use it when debugging the failed interrupted attempt.

4. Verify (quick sanity)

• An edit you made in step 2 shows up in the post-resume worker session logs/transcripts (the optimize session under the new workspace), and is absent from already-completed ones. Prompt edits are reflected in the worker session, not copied verbatim into the workspace as a file.
• The run continues under the original run id. Use checkpoint.json, lineage DBs, and telemetry/ for logical state; runner.log is an operational trace and may include failed pre-resume attempt output.
• To read the resulting population, see ../inspect-population/SKILL.md.

Gotchas

• Resume needs snapshots. They are written every iteration when loop.enable_resume=true (the default). A run launched with loop.enable_resume=false has no snapshots and cannot be resumed.
• Do not delete <run_root>/.snapshots/ or checkpoint.json — a missing snapshot makes resume fail hard.
• resume_from and import_from are mutually exclusive; setting both is an error.
• Remote wandb state is outside the local rollback contract. If wandb is enabled, treat it as remote operational logging, not as the source of truth for same-run resume.