kreuzcrawl
Crawl, scrape, and convert websites to Markdown using the local kreuzcrawl CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, mcp, serve), output formats (JSON + Markdown), browser fallback, and when to prefer the MCP server over shelling out.
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 1.6k
- Forks
- 491
01 前言 Front Matter
- name
- kreuzcrawl
- license
- MIT
- metadata
- { "author": "kreuzberg-dev", "version": "0.1.0", "repository": "https://github.com/kreuzberg-dev/kreuzcrawl" }
- description
- Crawl, scrape, and convert websites to Markdown using the local kreuzcrawl CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, mcp, serve), output formats (JSON + Markdown), browser fallback, and when to prefer the MCP server over shelling out.
03 SKILL.md 正文
Kreuzcrawl
Kreuzcrawl is a Rust-native web crawler and scraper. It fetches static HTML
with reqwest, falls back to headless Chrome when a page needs JS or trips a
WAF, and converts every result to clean Markdown via the built-in
HTML→Markdown engine.
Use this skill when the user wants to:
- Scrape a single URL to Markdown plus structured metadata.
- Crawl a site following links bounded by depth, page count, and concurrency.
- Enumerate URLs from sitemaps without paying for rendering.
- Drive a real browser (click, type, scroll) and capture the resulting DOM.
- Run the same operations from another agent harness via MCP tools.
Installation
The plugin shells out to a kreuzcrawl binary on PATH. Install one of:
brew install kreuzberg-dev/tap/kreuzcrawl
# or run without a persistent install (the CLI proxy package self-installs the binary):
npx @kreuzberg/kreuzcrawl-cli --help
uvx --from kreuzcrawl-cli kreuzcrawl --help
# or build from source:
cargo install --git https://github.com/kreuzberg-dev/kreuzcrawl kreuzcrawl-cli --features all
The serve and mcp subcommands are gated behind non-default cargo features
(api and mcp). The Homebrew tap is built with all features, so both
subcommands work out of the box. A from-source build must pass
--features mcp (and --features api for serve), or --features all, to
include them.
Verify:
kreuzcrawl --version
Headless fallback needs Chrome/Chromium reachable locally (chromiumoxide
launches it on demand). Skip the install if you only plan to use
--browser-mode never.
Command map
kreuzcrawl scrape <url> # single page → JSON or Markdown
kreuzcrawl crawl <url...> # follow links, BFS, depth-bounded
kreuzcrawl map <url> # enumerate URLs via sitemaps + link extraction
kreuzcrawl interact <url> # browser actions: click, type, scroll
kreuzcrawl mcp # MCP server (stdio) — auto-registered (`mcp` feature)
kreuzcrawl serve # REST API server (`api` feature)
Batch behaviour is built into crawl: pass multiple seed URLs and the engine
fans out via batch_crawl internally.
Shared flags
| Flag | Default | Notes |
|---|---|---|
--format |
json |
json or markdown. |
--timeout |
30000 |
Request timeout in milliseconds. |
--browser-mode |
auto |
auto, always, or never. |
--browser-endpoint |
— | Optional CDP ws:// or wss:// URL. |
--respect-robots-txt |
off | Pass to obey robots.txt. |
--config <json> |
— | Inline JSON or @file.json to override defaults. |
The --config flag accepts the full CrawlConfig schema. Anything you set
explicitly on the CLI overrides the corresponding JSON field.
Scrape a single page
kreuzcrawl scrape https://example.com --format markdown
JSON output (default) carries the rendered Markdown, page metadata
(PageMetadata), links by category, images, feeds, JSON-LD blocks, and
HTTP response metadata. Use Markdown output when piping into a file the user
will read.
See the scraping-html-to-markdown skill for the full flag surface.
Crawl a site
kreuzcrawl crawl https://example.com \
--depth 3 --max-pages 200 --concurrent 8 --rate-limit 250 \
--stay-on-domain --respect-robots-txt --format markdown
Crawling is BFS by default, bounded by --depth, --max-pages, and
--concurrent. Per-domain politeness is enforced by --rate-limit
(milliseconds between requests to the same origin).
See the crawling-a-site skill for the recommended defaults and the full
flag surface.
Map URLs
kreuzcrawl map https://example.com --limit 500 --search docs --format markdown
map reads sitemap.xml (and nested sitemaps), then falls back to link
extraction from the seed page. It does not render pages — use it to plan a
crawl or to feed URLs into another tool.
Browser interaction
kreuzcrawl interact https://example.com \
--actions '[{"type":"click","selector":"#load-more"},
{"type":"wait","milliseconds":500},
{"type":"scrape"}]'
Action types are click, type, press, scroll, wait, screenshot,
executeJs, and scrape (to wait for an element, use wait with a
selector field). The result wraps the final HTML under
interaction.final_html. See the automating-the-browser skill for the full
action schema and limits.
MCP server
When this plugin is installed in a Claude Code / Codex / Cursor / Gemini / opencode harness, the MCP server is auto-registered:
kreuzcrawl mcp
mcp is a stdio-transport server and takes no arguments. It requires a binary
built with the mcp feature (see Installation).
Prefer MCP tools over shelling out when both are available:
- Typed schemas surface argument errors before the call.
- Results stream back as structured tool output instead of stdout text.
- No
--formatjuggling — the harness pulls whatever shape it needs.
Fall back to the CLI when you need to script a pipeline, capture stderr, or chain with shell tools.
Headless fallback
In --browser-mode auto (default), the engine:
- Fetches statically via
reqwest. - Detects WAF blocks (8 vendors) and JS-only shells.
- Re-fetches through headless Chrome with a real fingerprint when needed.
Force the browser path with --browser-mode always when you already know
the page needs JS. Use --browser-mode never for hot loops where the cost
of a stray Chrome launch is unacceptable.
Point --browser-endpoint ws://host:9222/devtools/browser/<id> at an
already-running Chrome to skip the local launch.
See the headless-fallback skill for symptoms, costs, and external-CDP
patterns.
Output formats
| Mode | Use when |
|---|---|
json |
Downstream consumer needs metadata, links, images, etc. |
markdown |
Human reader or LLM-context payload. |
Markdown output skips metadata. If you need both, run with --format json
and read result.markdown.content.
Robots, rate limits, ethics
--respect-robots-txtis off by default; pass it for any crawl on a host you do not own.- The default
--rate-limit 200already produces a polite cadence; raise it for shared hosts. - Identify the crawler honestly via
--user-agent. Do not impersonate a browser unless the operator has approved it.
Cross-references
skills/crawling-a-site/SKILL.md— multi-page crawl with depth, page caps, concurrency, rate limits, and domain scoping.skills/scraping-html-to-markdown/SKILL.md— single-page rendering, the Markdown output shape, and common pitfalls.skills/mapping-urls/SKILL.md—map: sitemap + link URL discovery, filtering, and seeding a crawl.skills/automating-the-browser/SKILL.md—interact: the full scripted action schema, limits, and result shape.skills/serving-the-api/SKILL.md—serve: the Firecrawl-v1-compatible REST API server and its endpoints.skills/headless-fallback/SKILL.md— when and how to force the browser backend.
04 同仓库更多 Skills
worktable-review-with-annotations
Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.
worktable-find-and-synthesize-context
Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.
worktable-create-or-update-docs
Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.
worktable-create-or-manage-records
Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.
worktable-create-or-manage-interactive-html
Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.
worktable-collaborate-threads
Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.