mapping-urls
Use when the user wants the list of URLs on a site rather than the page content — sitemap analysis, link planning, or seeding another tool. Covers `kreuzcrawl map <url>` with `--limit`, `--search`, robots, output, and how it differs from a full crawl.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 1.6k
- Forks
- 491
01 前言 Front Matter
- name
- mapping-urls
- description
- Use when the user wants the list of URLs on a site rather than the page content — sitemap analysis, link planning, or seeding another tool. Covers `kreuzcrawl map <url>` with `--limit`, `--search`, robots, output, and how it differs from a full crawl.
03 SKILL.md 正文
Mapping URLs
kreuzcrawl map <url> discovers the URLs a site exposes without rendering or
extracting any page content. It reads sitemap.xml (including nested
sitemaps), then falls back to link extraction from the seed page. Use it to
plan a crawl, audit a site's surface, or feed a URL list into another tool.
Quick recipe
kreuzcrawl map https://example.com --limit 500 --search docs --format markdown
Markdown output prints one URL per line — convenient to pipe into a file or a
follow-up crawl. JSON output (default) returns a structured MapResult.
Flag surface
| Flag | Default | Purpose |
|---|---|---|
--limit |
— | Maximum number of URLs to return. Unbounded if unset. |
--search |
— | Case-insensitive substring filter on discovered URLs. |
--respect-robots-txt |
off | Honour robots.txt. Pass it for any third-party host. |
--format |
json |
json (full MapResult) or markdown (one URL per line). |
--timeout |
30000 |
Per-request timeout in ms. |
--browser-mode |
auto |
auto, always, never — see the headless-fallback skill. |
--browser-endpoint |
— | External CDP ws:// URL. |
--config |
— | Inline JSON or @file.json for the full CrawlConfig. |
map takes a single seed URL positionally. There is no --depth or
--max-pages here — those bound a crawl, not a map. Scope is the seed host's
sitemaps plus links found on the seed page; bound the result with --limit and
narrow it with --search.
How discovery works
- Fetch and parse
sitemap.xml, following nested<sitemapindex>entries. - If no sitemap (or a thin one), extract links from the seed page's HTML.
- Apply the
--searchsubstring filter (case-insensitive), then--limit.
No page bodies are rendered, so a map of hundreds of URLs returns in seconds —
far cheaper than crawling. In --browser-mode auto the seed fetch still falls
back to headless Chrome if the seed page is a JS shell that hides its links;
pass --browser-mode never to keep it static-only.
Output
Markdown mode
https://example.com/
https://example.com/docs/
https://example.com/docs/getting-started
https://example.com/blog/post-one
JSON mode
Top-level MapResult with a urls array; each entry carries the discovered
url. Read result.urls[i].url for each string when scripting.
kreuzcrawl map https://example.com --format json | jq -r '.urls[].url'
Common patterns
Discover then crawl a subsection
kreuzcrawl map https://example.com --search /docs/ --format markdown > urls.txt
Feed the filtered list into a bounded crawl, or scrape individual entries.
Audit a third-party site politely
kreuzcrawl map https://unknown.example --respect-robots-txt --limit 200
When to reach for crawl instead
If the user needs the page content (Markdown, metadata, tables) rather than
just the URL list, use kreuzcrawl crawl — see the crawling-a-site skill.
Reach for map first when the goal is enumeration, planning, or seeding.
04 同仓库更多 Skills
worktable-review-with-annotations
Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.
worktable-find-and-synthesize-context
Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.
worktable-create-or-update-docs
Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.
worktable-create-or-manage-records
Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.
worktable-create-or-manage-interactive-html
Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.
worktable-collaborate-threads
Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.