crawling-a-site
Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 1.6k
- Forks
- 491
01 前言 Front Matter
- name
- crawling-a-site
- description
- Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.
03 SKILL.md 正文
<!-- AI-RULEZ :: GENERATED FILE — DO NOT EDIT Content-Hash: blake3:87e62f8235715c6d9ff820d40d26b521d5d0e0fb9888f7d04589c7429d28da14 Source-Hash: blake3:b5689383a3914da8e3cc6ad3614356ae7f98c90cf99cb1b4b47be38456ff7a7c Schema-Version: v1 -->
Crawling a site
Reach for crawlberg crawl when one URL is not enough — the user wants
the docs site, the blog, the marketing pages, or the whole domain.
Quick recipe
crawlberg crawl https://example.com \
--depth 3 \
--max-pages 200 \
--concurrent 8 \
--rate-limit 250 \
--stay-on-domain \
--respect-robots-txt \
--format markdown
Defaults you should usually override:
--depth 2is shallow — set it explicitly.--max-pagesis unbounded by default; cap it for any unknown site.--concurrent 10is aggressive for small hosts; drop to 4-8 for third-party sites.
Flag surface
| Flag | Default | Purpose |
|---|---|---|
--depth, -d |
2 |
Maximum hop count from the seed URL. |
--max-pages, -n |
— | Hard cap on pages fetched. Set this on any unknown site. |
--concurrent, -c |
10 |
Parallel in-flight requests. |
--rate-limit |
200 |
Milliseconds between requests to the same origin. |
--stay-on-domain |
off | Skip links that leave the seed domain. |
--respect-robots-txt |
off | Honour robots.txt. Pass it for any third-party host. |
--proxy |
— | HTTP, HTTPS, or SOCKS5 proxy URL. |
--user-agent |
— | Override the request UA. Be honest. |
--timeout |
30000 |
Per-request timeout in ms. |
--browser-mode |
auto |
auto, always, never — see the headless-fallback skill. |
--browser-endpoint |
— | External CDP ws:// URL. |
--format |
json |
json or markdown. |
--config |
— | Inline JSON or @file.json for the full CrawlConfig. |
Multiple seed URLs are accepted positionally — the engine fans out with
batch_crawl and aggregates results.
When to pick which flags
Docs sites you own
crawlberg crawl https://docs.example.com \
--depth 5 --max-pages 1000 --concurrent 16 --rate-limit 100 \
--stay-on-domain --format markdown > docs.md
Higher concurrency and lower rate limits are fine on infrastructure you control.
Third-party sites
crawlberg crawl https://blog.unknown.example \
--depth 2 --max-pages 50 --concurrent 4 --rate-limit 500 \
--stay-on-domain --respect-robots-txt --format markdown
Stay shallow, cap pages, throttle hard, obey robots.
Multi-seed batch
crawlberg crawl \
https://example.com/blog \
https://example.com/docs \
https://example.com/pricing \
--depth 2 --max-pages 100 --stay-on-domain --format json
JSON output for batch is an array of { seed_url, result } entries — each
result is a full crawl payload or { error: ... }.
Output
Markdown mode
---
URL: https://example.com/page-one
---
# Page One
… markdown content …
---
URL: https://example.com/page-two
---
…
JSON mode
Top-level CrawlResult with pages: [...]. Each page carries the rendered
Markdown plus metadata, links, images, JSON-LD, and HTTP response info. Read
result.pages[i].markdown.content for the Markdown string.
Politeness checklist
- Pass
--respect-robots-txton every third-party crawl. - Cap
--max-pages— a runaway BFS can issue tens of thousands of requests. - Bump
--rate-limitfor hosts that show signs of stress (5xx, slowdowns). - Identify yourself via
--user-agent crawlberg (contact@example.com).
Common pitfalls
- No pages returned. The seed page may be JS-only — the engine falls
back to headless automatically in
--browser-mode auto, butnevermode will silently produce an empty crawl. Re-run with--browser-mode alwaysor check the headless-fallback skill. - Crawl leaves the domain. Pass
--stay-on-domain. Combine withallow_subdomains: truein--configJSON to include subdomains. - Slow crawl. The default rate limit is 200 ms per origin — multiple
seed URLs on the same host still share the bucket. Spread seeds across
hosts or raise
--concurrentfor unrelated origins. - Memory growth. Each page carries full Markdown plus structured data.
Stream JSON output to a file rather than holding it in memory; set
--max-pagesaggressively if downstream cannot keep up.
When to reach for map instead
If the user only needs the list of URLs (sitemap analysis, link planning,
seeding another tool), use crawlberg map <url> — it skips rendering and
returns a flat MapResult with hundreds of URLs in seconds.
04 同仓库更多 Skills
worktable-review-with-annotations
Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.
worktable-find-and-synthesize-context
Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.
worktable-create-or-update-docs
Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.
worktable-create-or-manage-records
Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.
worktable-create-or-manage-interactive-html
Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.
worktable-collaborate-threads
Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.