AGENT SKILL

crawling-a-site

Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.

License 许可
Apache-2.0
最近更新
2026-11-14
Stars
★ 1.6k
Forks
491

01 前言 Front Matter

name
crawling-a-site
description
Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.

03 SKILL.md 正文

<!-- AI-RULEZ :: GENERATED FILE — DO NOT EDIT Content-Hash: blake3:87e62f8235715c6d9ff820d40d26b521d5d0e0fb9888f7d04589c7429d28da14 Source-Hash: blake3:b5689383a3914da8e3cc6ad3614356ae7f98c90cf99cb1b4b47be38456ff7a7c Schema-Version: v1 -->

Crawling a site

Reach for crawlberg crawl when one URL is not enough — the user wants the docs site, the blog, the marketing pages, or the whole domain.

Quick recipe

crawlberg crawl https://example.com \
  --depth 3 \
  --max-pages 200 \
  --concurrent 8 \
  --rate-limit 250 \
  --stay-on-domain \
  --respect-robots-txt \
  --format markdown

Defaults you should usually override:

  • --depth 2 is shallow — set it explicitly.
  • --max-pages is unbounded by default; cap it for any unknown site.
  • --concurrent 10 is aggressive for small hosts; drop to 4-8 for third-party sites.

Flag surface

Flag Default Purpose
--depth, -d 2 Maximum hop count from the seed URL.
--max-pages, -n — Hard cap on pages fetched. Set this on any unknown site.
--concurrent, -c 10 Parallel in-flight requests.
--rate-limit 200 Milliseconds between requests to the same origin.
--stay-on-domain off Skip links that leave the seed domain.
--respect-robots-txt off Honour robots.txt. Pass it for any third-party host.
--proxy — HTTP, HTTPS, or SOCKS5 proxy URL.
--user-agent — Override the request UA. Be honest.
--timeout 30000 Per-request timeout in ms.
--browser-mode auto auto, always, never — see the headless-fallback skill.
--browser-endpoint — External CDP ws:// URL.
--format json json or markdown.
--config — Inline JSON or @file.json for the full CrawlConfig.

Multiple seed URLs are accepted positionally — the engine fans out with batch_crawl and aggregates results.

When to pick which flags

Docs sites you own

crawlberg crawl https://docs.example.com \
  --depth 5 --max-pages 1000 --concurrent 16 --rate-limit 100 \
  --stay-on-domain --format markdown > docs.md

Higher concurrency and lower rate limits are fine on infrastructure you control.

Third-party sites

crawlberg crawl https://blog.unknown.example \
  --depth 2 --max-pages 50 --concurrent 4 --rate-limit 500 \
  --stay-on-domain --respect-robots-txt --format markdown

Stay shallow, cap pages, throttle hard, obey robots.

Multi-seed batch

crawlberg crawl \
  https://example.com/blog \
  https://example.com/docs \
  https://example.com/pricing \
  --depth 2 --max-pages 100 --stay-on-domain --format json

JSON output for batch is an array of { seed_url, result } entries — each result is a full crawl payload or { error: ... }.

Output

Markdown mode

---
URL: https://example.com/page-one
---
# Page One

… markdown content …

---
URL: https://example.com/page-two
---
…

JSON mode

Top-level CrawlResult with pages: [...]. Each page carries the rendered Markdown plus metadata, links, images, JSON-LD, and HTTP response info. Read result.pages[i].markdown.content for the Markdown string.

Politeness checklist

  • Pass --respect-robots-txt on every third-party crawl.
  • Cap --max-pages — a runaway BFS can issue tens of thousands of requests.
  • Bump --rate-limit for hosts that show signs of stress (5xx, slowdowns).
  • Identify yourself via --user-agent crawlberg (contact@example.com).

Common pitfalls

  • No pages returned. The seed page may be JS-only — the engine falls back to headless automatically in --browser-mode auto, but never mode will silently produce an empty crawl. Re-run with --browser-mode always or check the headless-fallback skill.
  • Crawl leaves the domain. Pass --stay-on-domain. Combine with allow_subdomains: true in --config JSON to include subdomains.
  • Slow crawl. The default rate limit is 200 ms per origin — multiple seed URLs on the same host still share the bucket. Spread seeds across hosts or raise --concurrent for unrelated origins.
  • Memory growth. Each page carries full Markdown plus structured data. Stream JSON output to a file rather than holding it in memory; set --max-pages aggressively if downstream cannot keep up.

When to reach for map instead

If the user only needs the list of URLs (sitemap analysis, link planning, seeding another tool), use crawlberg map <url> — it skips rendering and returns a flat MapResult with hundreds of URLs in seconds.

04 同仓库更多 Skills

hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-review-with-annotations

Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-find-and-synthesize-context

Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-update-docs

Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-records

Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-interactive-html

Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-collaborate-threads

Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.

★ 1.6k