AGENT SKILL

mapping-urls

Use when the user wants the list of URLs on a site rather than the page content — sitemap analysis, link planning, or seeding another tool. Covers `kreuzcrawl map <url>` with `--limit`, `--search`, robots, output, and how it differs from a full crawl.

License 许可
Apache-2.0
最近更新
2026-11-14
Stars
★ 1.6k
Forks
491

01 前言 Front Matter

name
mapping-urls
description
Use when the user wants the list of URLs on a site rather than the page content — sitemap analysis, link planning, or seeding another tool. Covers `kreuzcrawl map <url>` with `--limit`, `--search`, robots, output, and how it differs from a full crawl.

03 SKILL.md 正文

Mapping URLs

kreuzcrawl map <url> discovers the URLs a site exposes without rendering or extracting any page content. It reads sitemap.xml (including nested sitemaps), then falls back to link extraction from the seed page. Use it to plan a crawl, audit a site's surface, or feed a URL list into another tool.

Quick recipe

kreuzcrawl map https://example.com --limit 500 --search docs --format markdown

Markdown output prints one URL per line — convenient to pipe into a file or a follow-up crawl. JSON output (default) returns a structured MapResult.

Flag surface

Flag Default Purpose
--limit — Maximum number of URLs to return. Unbounded if unset.
--search — Case-insensitive substring filter on discovered URLs.
--respect-robots-txt off Honour robots.txt. Pass it for any third-party host.
--format json json (full MapResult) or markdown (one URL per line).
--timeout 30000 Per-request timeout in ms.
--browser-mode auto auto, always, never — see the headless-fallback skill.
--browser-endpoint — External CDP ws:// URL.
--config — Inline JSON or @file.json for the full CrawlConfig.

map takes a single seed URL positionally. There is no --depth or --max-pages here — those bound a crawl, not a map. Scope is the seed host's sitemaps plus links found on the seed page; bound the result with --limit and narrow it with --search.

How discovery works

  1. Fetch and parse sitemap.xml, following nested <sitemapindex> entries.
  2. If no sitemap (or a thin one), extract links from the seed page's HTML.
  3. Apply the --search substring filter (case-insensitive), then --limit.

No page bodies are rendered, so a map of hundreds of URLs returns in seconds — far cheaper than crawling. In --browser-mode auto the seed fetch still falls back to headless Chrome if the seed page is a JS shell that hides its links; pass --browser-mode never to keep it static-only.

Output

Markdown mode

https://example.com/
https://example.com/docs/
https://example.com/docs/getting-started
https://example.com/blog/post-one

JSON mode

Top-level MapResult with a urls array; each entry carries the discovered url. Read result.urls[i].url for each string when scripting.

kreuzcrawl map https://example.com --format json | jq -r '.urls[].url'

Common patterns

Discover then crawl a subsection

kreuzcrawl map https://example.com --search /docs/ --format markdown > urls.txt

Feed the filtered list into a bounded crawl, or scrape individual entries.

Audit a third-party site politely

kreuzcrawl map https://unknown.example --respect-robots-txt --limit 200

When to reach for crawl instead

If the user needs the page content (Markdown, metadata, tables) rather than just the URL list, use kreuzcrawl crawl — see the crawling-a-site skill. Reach for map first when the goal is enumeration, planning, or seeding.

04 同仓库更多 Skills

hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-review-with-annotations

Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-find-and-synthesize-context

Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-update-docs

Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-records

Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-interactive-html

Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-collaborate-threads

Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.

★ 1.6k