run-deep-swe
Run reproducible DeepSWE coding-agent benchmark evaluations through OpenRouter and mini-swe-agent.
- License 许可
- MIT
- 最近更新
- 2026-10-25
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- run-deep-swe
- risk
- critical
- tags
- [ "benchmark", "deepswe", "openrouter", "evaluation" ]
- tools
- [ "claude", "codex" ]
- author
- davidondrej
- source
- community
- license
- MIT
- category
- agent-evaluation
- date_added
- 2026-07-07
- description
- Run reproducible DeepSWE coding-agent benchmark evaluations through OpenRouter and mini-swe-agent.
- source_repo
- davidondrej/skills
- source_type
- community
- license_source
- https://github.com/davidondrej/skills/blob/main/LICENSE
- disable-model-invocation
- true
03 SKILL.md 正文
Run DeepSWE via OpenRouter
When to Use
- Use when the user wants to benchmark a model on DeepSWE or mini-swe-agent tasks.
- Use when you need a reproducible coding-agent evaluation plan and output artifacts.
DeepSWE (deepswe.datacurve.ai) is a 113-task Harbor-compatible coding-agent benchmark. It runs via Pier (Harbor fork) driving mini-swe-agent (model-agnostic). Any model reachable through OpenRouter can be scored.
Prerequisites — state-check first
which uv git docker || echo "MISSING: install uv, git, docker"
docker info >/dev/null 2>&1 || echo "MISSING: Docker daemon not running (Pier's default sandbox)"
echo "OPENROUTER_API_KEY set? ${OPENROUTER_API_KEY:+YES}"
Docker must be running — Pier sandboxes each task in Docker by default (--env modal for cloud instead).
OPENROUTER_API_KEY must already be present in the environment. If it is unset,
ask the user to configure their preferred secret-management path; do not read
shell startup files, print secrets, or invent a key.
Setup
git clone https://github.com/datacurve-ai/deep-swe && cd deep-swe
uv tool install datacurve-pier # PyPI (preferred)
# or: uv tool install git+https://github.com/datacurve-ai/pier
# pier bundles mini-swe-agent as the --agent driver
Run all pier commands from inside deep-swe/, using relative -p tasks/....
OpenRouter wiring (the part the docs don't spell out)
mini-swe-agent has a native OpenRouter model class. Both routes below use OPENROUTER_API_KEY and the OpenRouter slug (vendor/model, e.g. minimax/minimax-m3):
Route A — native OpenRouter class (preferred, hits openrouter.ai/api/v1 directly):
pier run -p deep-swe/tasks --agent mini-swe-agent \
--model minimax/minimax-m3 --model-class openrouter
Route B — LiteLLM provider prefix (fallback; same key):
pier run -p deep-swe/tasks --agent mini-swe-agent \
--model openrouter/minimax/minimax-m3
Notes:
- Slug = the exact OpenRouter slug. Verify it at openrouter.ai/models before running.
- Free/zero-cost models: OpenRouter cost tracking can error. Set
export MSWEA_COST_TRACKING=ignore_errors. - Flag spelling can vary by version — confirm with
pier run --helpandmini --help.
Smoke test FIRST (1 task — do this before any full run)
Always validate end-to-end wiring on a single task before spending tokens on the corpus:
pier run -p deep-swe/tasks/<task-id> --agent mini-swe-agent \
--model minimax/minimax-m3 --model-class openrouter
# list available task ids:
ls deep-swe/tasks
Pass criteria: run completes, model returns actions (not auth/format errors), a score/trajectory is emitted. If it 401s → key wrong. If "provider not provided"/"model not mapped" → fix slug or switch route.
Subset run (deterministic sample)
pier run -p deep-swe/tasks --agent mini-swe-agent \
--model minimax/minimax-m3 --model-class openrouter \
--n-tasks 10 --sample-seed 0
Full 113-task corpus (costs tokens + time — confirm with user first)
pier run -p deep-swe/tasks --agent mini-swe-agent \
--model minimax/minimax-m3 --model-class openrouter
# add `--env modal` to run in parallel Modal sandboxes (needs Modal configured)
Output & leaderboard
- Trials land in
jobs/<run>/<trial_id>/. Inspect withpier view jobs/<run>,pier analyze jobs/<run>, orpier critique run jobs/<run>. - Report: the exact command used, pass/fail, score, and any blockers.
- Submit results for the official leaderboard to: <email-address>
Failure modes
| Symptom | Cause | Fix |
|---|---|---|
| HTTP 401 | bad/missing key | re-export OPENROUTER_API_KEY |
| "LLM Provider NOT provided" | missing slug prefix | use Route B openrouter/... or Route A with --model-class openrouter |
| "model isn't mapped"/cost error | unknown cost for model | export MSWEA_COST_TRACKING=ignore_errors |
| unknown flag | version drift | check pier run --help |
Limitations
- Adapted from
davidondrej/skills; verify local paths, tools, credentials, and agent features before acting. - For commands, remote access, scheduling, browser automation, or file-changing workflows, get explicit user approval and confirm the target environment first.
04 同仓库更多 Skills
zero-trust
Implement zero-trust network architecture. Configure identity-based access, micro-segmentation, and continuous verification. Use when implementing modern security architectures.
zcode-delegate
Delegate coding tasks to the Z.AI ZCode CLI only when the user explicitly requests it, while the orchestrator retains review and landing responsibility.
youtube-transcript
Fetch YouTube transcripts through DeepAPI or local fallback tooling and save clean text output.
youtube-transcript-skills
Fetch YouTube video transcripts, search videos/channels, browse channels, and extract playlists via the getyoutubetranscript.com API - free tier, no card required.
youtube-notetaker
Turn YouTube talks into local study notes with slides, transcripts, editable annotations, and a markdown-backed viewer.
youtube-full
Fetch YouTube transcripts, search videos, browse channels, and extract playlists via TranscriptAPI — no yt-dlp, no Google API key, works from any cloud server.