[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MHYT4VDHX4CGPRYN2Y5WZE":3,"skill-same:01M6MHYT4VDHX4CGPRYN2Y5WZE":70},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":41,"linkOnly":42,"markdownContent":43,"markdownBody":44,"frontMatter":45,"refName":46,"commitSha":47,"skillDirectory":48,"forksCount":49,"repositoryHtmlUrl":50,"repositoryHomepageUrl":51,"ownerHtmlUrl":52,"ownerBlogUrl":53,"ownerDescription":54,"updatedAt":55,"install":56,"resources":69},"01M6MHYT4VDHX4CGPRYN2Y5WZE","crawling-a-site",null,"Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fkreuzberg-dev\u002Fplugins\u002Fplugins\u002Fcrawlberg\u002Fskills\u002Fcrawling-a-site\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40],"site","crawling a site","crawling","rate limiting","when","depth","crawlberg","rate","links","with","output","as","concurrency","across","domain","crawl","selection","use","scoping","capture","page","markdown","2026-11-10 10:12:30",false,"---\nname: crawling-a-site\ndescription: >-\n  Use when the user wants to follow links across a domain and capture every\n  reachable page as Markdown. Covers `crawlberg crawl` with depth, page\n  caps, concurrency, rate limiting, domain scoping, robots, and output\n  selection.\n---\n\n\u003C!--\nAI-RULEZ :: GENERATED FILE — DO NOT EDIT\nContent-Hash: blake3:87e62f8235715c6d9ff820d40d26b521d5d0e0fb9888f7d04589c7429d28da14\nSource-Hash: blake3:b5689383a3914da8e3cc6ad3614356ae7f98c90cf99cb1b4b47be38456ff7a7c\nSchema-Version: v1\n-->\n\n# Crawling a site\n\nReach for `crawlberg crawl` when one URL is not enough — the user wants\nthe docs site, the blog, the marketing pages, or the whole domain.\n\n## Quick recipe\n\n```bash\ncrawlberg crawl https:\u002F\u002Fexample.com \\\n  --depth 3 \\\n  --max-pages 200 \\\n  --concurrent 8 \\\n  --rate-limit 250 \\\n  --stay-on-domain \\\n  --respect-robots-txt \\\n  --format markdown\n```\n\nDefaults you should usually override:\n\n- `--depth 2` is shallow — set it explicitly.\n- `--max-pages` is unbounded by default; cap it for any unknown site.\n- `--concurrent 10` is aggressive for small hosts; drop to 4-8 for\n  third-party sites.\n\n## Flag surface\n\n| Flag                    | Default | Purpose                                                    |\n| ----------------------- | ------- | ---------------------------------------------------------- |\n| `--depth`, `-d`         | `2`     | Maximum hop count from the seed URL.                       |\n| `--max-pages`, `-n`     | —       | Hard cap on pages fetched. Set this on any unknown site.   |\n| `--concurrent`, `-c`    | `10`    | Parallel in-flight requests.                               |\n| `--rate-limit`          | `200`   | Milliseconds between requests to the same origin.          |\n| `--stay-on-domain`      | off     | Skip links that leave the seed domain.                     |\n| `--respect-robots-txt`  | off     | Honour `robots.txt`. Pass it for any third-party host.     |\n| `--proxy`               | —       | HTTP, HTTPS, or SOCKS5 proxy URL.                          |\n| `--user-agent`          | —       | Override the request UA. Be honest.                        |\n| `--timeout`             | `30000` | Per-request timeout in ms.                                 |\n| `--browser-mode`        | `auto`  | `auto`, `always`, `never` — see the headless-fallback skill. |\n| `--browser-endpoint`    | —       | External CDP `ws:\u002F\u002F` URL.                                  |\n| `--format`              | `json`  | `json` or `markdown`.                                      |\n| `--config`              | —       | Inline JSON or `@file.json` for the full `CrawlConfig`.    |\n\nMultiple seed URLs are accepted positionally — the engine fans out with\n`batch_crawl` and aggregates results.\n\n## When to pick which flags\n\n### Docs sites you own\n\n```bash\ncrawlberg crawl https:\u002F\u002Fdocs.example.com \\\n  --depth 5 --max-pages 1000 --concurrent 16 --rate-limit 100 \\\n  --stay-on-domain --format markdown > docs.md\n```\n\nHigher concurrency and lower rate limits are fine on infrastructure you\ncontrol.\n\n### Third-party sites\n\n```bash\ncrawlberg crawl https:\u002F\u002Fblog.unknown.example \\\n  --depth 2 --max-pages 50 --concurrent 4 --rate-limit 500 \\\n  --stay-on-domain --respect-robots-txt --format markdown\n```\n\nStay shallow, cap pages, throttle hard, obey robots.\n\n### Multi-seed batch\n\n```bash\ncrawlberg crawl \\\n  https:\u002F\u002Fexample.com\u002Fblog \\\n  https:\u002F\u002Fexample.com\u002Fdocs \\\n  https:\u002F\u002Fexample.com\u002Fpricing \\\n  --depth 2 --max-pages 100 --stay-on-domain --format json\n```\n\nJSON output for batch is an array of `{ seed_url, result }` entries — each\n`result` is a full crawl payload or `{ error: ... }`.\n\n## Output\n\n### Markdown mode\n\n```text\n---\nURL: https:\u002F\u002Fexample.com\u002Fpage-one\n---\n# Page One\n\n… markdown content …\n\n---\nURL: https:\u002F\u002Fexample.com\u002Fpage-two\n---\n…\n```\n\n### JSON mode\n\nTop-level `CrawlResult` with `pages: [...]`. Each page carries the rendered\nMarkdown plus metadata, links, images, JSON-LD, and HTTP response info. Read\n`result.pages[i].markdown.content` for the Markdown string.\n\n## Politeness checklist\n\n- Pass `--respect-robots-txt` on every third-party crawl.\n- Cap `--max-pages` — a runaway BFS can issue tens of thousands of requests.\n- Bump `--rate-limit` for hosts that show signs of stress (5xx, slowdowns).\n- Identify yourself via `--user-agent crawlberg (contact@example.com)`.\n\n## Common pitfalls\n\n- **No pages returned.** The seed page may be JS-only — the engine falls\n  back to headless automatically in `--browser-mode auto`, but `never` mode\n  will silently produce an empty crawl. Re-run with `--browser-mode always`\n  or check the headless-fallback skill.\n- **Crawl leaves the domain.** Pass `--stay-on-domain`. Combine with\n  `allow_subdomains: true` in `--config` JSON to include subdomains.\n- **Slow crawl.** The default rate limit is 200 ms per origin — multiple\n  seed URLs on the same host still share the bucket. Spread seeds across\n  hosts or raise `--concurrent` for unrelated origins.\n- **Memory growth.** Each page carries full Markdown plus structured data.\n  Stream JSON output to a file rather than holding it in memory; set\n  `--max-pages` aggressively if downstream cannot keep up.\n\n## When to reach for `map` instead\n\nIf the user only needs the list of URLs (sitemap analysis, link planning,\nseeding another tool), use `crawlberg map \u003Curl>` — it skips rendering and\nreturns a flat `MapResult` with hundreds of URLs in seconds.\n","\u003C!--\nAI-RULEZ :: GENERATED FILE — DO NOT EDIT\nContent-Hash: blake3:87e62f8235715c6d9ff820d40d26b521d5d0e0fb9888f7d04589c7429d28da14\nSource-Hash: blake3:b5689383a3914da8e3cc6ad3614356ae7f98c90cf99cb1b4b47be38456ff7a7c\nSchema-Version: v1\n-->\n\n# Crawling a site\n\nReach for `crawlberg crawl` when one URL is not enough — the user wants\nthe docs site, the blog, the marketing pages, or the whole domain.\n\n## Quick recipe\n\n```bash\ncrawlberg crawl https:\u002F\u002Fexample.com \\\n  --depth 3 \\\n  --max-pages 200 \\\n  --concurrent 8 \\\n  --rate-limit 250 \\\n  --stay-on-domain \\\n  --respect-robots-txt \\\n  --format markdown\n```\n\nDefaults you should usually override:\n\n- `--depth 2` is shallow — set it explicitly.\n- `--max-pages` is unbounded by default; cap it for any unknown site.\n- `--concurrent 10` is aggressive for small hosts; drop to 4-8 for\n  third-party sites.\n\n## Flag surface\n\n| Flag                    | Default | Purpose                                                    |\n| ----------------------- | ------- | ---------------------------------------------------------- |\n| `--depth`, `-d`         | `2`     | Maximum hop count from the seed URL.                       |\n| `--max-pages`, `-n`     | —       | Hard cap on pages fetched. Set this on any unknown site.   |\n| `--concurrent`, `-c`    | `10`    | Parallel in-flight requests.                               |\n| `--rate-limit`          | `200`   | Milliseconds between requests to the same origin.          |\n| `--stay-on-domain`      | off     | Skip links that leave the seed domain.                     |\n| `--respect-robots-txt`  | off     | Honour `robots.txt`. Pass it for any third-party host.     |\n| `--proxy`               | —       | HTTP, HTTPS, or SOCKS5 proxy URL.                          |\n| `--user-agent`          | —       | Override the request UA. Be honest.                        |\n| `--timeout`             | `30000` | Per-request timeout in ms.                                 |\n| `--browser-mode`        | `auto`  | `auto`, `always`, `never` — see the headless-fallback skill. |\n| `--browser-endpoint`    | —       | External CDP `ws:\u002F\u002F` URL.                                  |\n| `--format`              | `json`  | `json` or `markdown`.                                      |\n| `--config`              | —       | Inline JSON or `@file.json` for the full `CrawlConfig`.    |\n\nMultiple seed URLs are accepted positionally — the engine fans out with\n`batch_crawl` and aggregates results.\n\n## When to pick which flags\n\n### Docs sites you own\n\n```bash\ncrawlberg crawl https:\u002F\u002Fdocs.example.com \\\n  --depth 5 --max-pages 1000 --concurrent 16 --rate-limit 100 \\\n  --stay-on-domain --format markdown > docs.md\n```\n\nHigher concurrency and lower rate limits are fine on infrastructure you\ncontrol.\n\n### Third-party sites\n\n```bash\ncrawlberg crawl https:\u002F\u002Fblog.unknown.example \\\n  --depth 2 --max-pages 50 --concurrent 4 --rate-limit 500 \\\n  --stay-on-domain --respect-robots-txt --format markdown\n```\n\nStay shallow, cap pages, throttle hard, obey robots.\n\n### Multi-seed batch\n\n```bash\ncrawlberg crawl \\\n  https:\u002F\u002Fexample.com\u002Fblog \\\n  https:\u002F\u002Fexample.com\u002Fdocs \\\n  https:\u002F\u002Fexample.com\u002Fpricing \\\n  --depth 2 --max-pages 100 --stay-on-domain --format json\n```\n\nJSON output for batch is an array of `{ seed_url, result }` entries — each\n`result` is a full crawl payload or `{ error: ... }`.\n\n## Output\n\n### Markdown mode\n\n```text\n---\nURL: https:\u002F\u002Fexample.com\u002Fpage-one\n---\n# Page One\n\n… markdown content …\n\n---\nURL: https:\u002F\u002Fexample.com\u002Fpage-two\n---\n…\n```\n\n### JSON mode\n\nTop-level `CrawlResult` with `pages: [...]`. Each page carries the rendered\nMarkdown plus metadata, links, images, JSON-LD, and HTTP response info. Read\n`result.pages[i].markdown.content` for the Markdown string.\n\n## Politeness checklist\n\n- Pass `--respect-robots-txt` on every third-party crawl.\n- Cap `--max-pages` — a runaway BFS can issue tens of thousands of requests.\n- Bump `--rate-limit` for hosts that show signs of stress (5xx, slowdowns).\n- Identify yourself via `--user-agent crawlberg (contact@example.com)`.\n\n## Common pitfalls\n\n- **No pages returned.** The seed page may be JS-only — the engine falls\n  back to headless automatically in `--browser-mode auto`, but `never` mode\n  will silently produce an empty crawl. Re-run with `--browser-mode always`\n  or check the headless-fallback skill.\n- **Crawl leaves the domain.** Pass `--stay-on-domain`. Combine with\n  `allow_subdomains: true` in `--config` JSON to include subdomains.\n- **Slow crawl.** The default rate limit is 200 ms per origin — multiple\n  seed URLs on the same host still share the bucket. Spread seeds across\n  hosts or raise `--concurrent` for unrelated origins.\n- **Memory growth.** Each page carries full Markdown plus structured data.\n  Stream JSON output to a file rather than holding it in memory; set\n  `--max-pages` aggressively if downstream cannot keep up.\n\n## When to reach for `map` instead\n\nIf the user only needs the list of URLs (sitemap analysis, link planning,\nseeding another tool), use `crawlberg map \u003Curl>` — it skips rendering and\nreturns a flat `MapResult` with hundreds of URLs in seconds.\n",{"name":5,"description":7},"main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002Fkreuzberg-dev\u002Fplugins\u002Fplugins\u002Fcrawlberg\u002Fskills\u002Fcrawling-a-site",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":57,"prompt":67,"zipAvailable":68},[58,61,64],{"tool":59,"command":60},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill crawling-a-site",{"tool":62,"command":63},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill crawling-a-site",{"tool":65,"command":66},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill crawling-a-site","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fkreuzberg-dev\u002Fplugins\u002Fplugins\u002Fcrawlberg\u002Fskills\u002Fcrawling-a-site\u002FSKILL.md",true,[],[71,81,89,97,105,113],{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":75,"licenseFileUrl":76,"sourceHtmlUrl":77,"categories":78,"keywords":79,"publishedAt":80},"01M7DV332YQMGD8WTRJ9DZQXTB","worktable-review-with-annotations","Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.","MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":75,"licenseFileUrl":85,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":80},"01M7DV2YGHCZVBEB1AHYB6CGHE","worktable-find-and-synthesize-context","Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FSKILL.md",[],[],{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":75,"licenseFileUrl":93,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":80},"01M7DV2TQN1H11ME2SAF9MXN7V","worktable-create-or-update-docs","Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FSKILL.md",[],[],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":75,"licenseFileUrl":101,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":80},"01M7DV2NE452NEA8TXATKSNAJH","worktable-create-or-manage-records","Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FSKILL.md",[],[],{"skillNo":106,"name":107,"nameZh":6,"description":108,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":75,"licenseFileUrl":109,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":80},"01M7DV2HENRBR2KSCVM064WH0Y","worktable-create-or-manage-interactive-html","Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FSKILL.md",[],[],{"skillNo":114,"name":115,"nameZh":6,"description":116,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":75,"licenseFileUrl":117,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":80},"01M7DV2CADR5R7RQFFYPNAGR9Y","worktable-collaborate-threads","Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FSKILL.md",[],[]]