AGENT SKILL

kreuzcrawl

Crawl, scrape, and convert websites to Markdown using the local kreuzcrawl CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, mcp, serve), output formats (JSON + Markdown), browser fallback, and when to prefer the MCP server over shelling out.

License 许可
MIT
最近更新
2026-11-14
Stars
★ 1.6k
Forks
491

01 前言 Front Matter

name
kreuzcrawl
license
MIT
metadata
{ "author": "kreuzberg-dev", "version": "0.1.0", "repository": "https://github.com/kreuzberg-dev/kreuzcrawl" }
description
Crawl, scrape, and convert websites to Markdown using the local kreuzcrawl CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, mcp, serve), output formats (JSON + Markdown), browser fallback, and when to prefer the MCP server over shelling out.

03 SKILL.md 正文

Kreuzcrawl

Kreuzcrawl is a Rust-native web crawler and scraper. It fetches static HTML with reqwest, falls back to headless Chrome when a page needs JS or trips a WAF, and converts every result to clean Markdown via the built-in HTML→Markdown engine.

Use this skill when the user wants to:

  • Scrape a single URL to Markdown plus structured metadata.
  • Crawl a site following links bounded by depth, page count, and concurrency.
  • Enumerate URLs from sitemaps without paying for rendering.
  • Drive a real browser (click, type, scroll) and capture the resulting DOM.
  • Run the same operations from another agent harness via MCP tools.

Installation

The plugin shells out to a kreuzcrawl binary on PATH. Install one of:

brew install kreuzberg-dev/tap/kreuzcrawl
# or run without a persistent install (the CLI proxy package self-installs the binary):
npx @kreuzberg/kreuzcrawl-cli --help
uvx --from kreuzcrawl-cli kreuzcrawl --help
# or build from source:
cargo install --git https://github.com/kreuzberg-dev/kreuzcrawl kreuzcrawl-cli --features all

The serve and mcp subcommands are gated behind non-default cargo features (api and mcp). The Homebrew tap is built with all features, so both subcommands work out of the box. A from-source build must pass --features mcp (and --features api for serve), or --features all, to include them.

Verify:

kreuzcrawl --version

Headless fallback needs Chrome/Chromium reachable locally (chromiumoxide launches it on demand). Skip the install if you only plan to use --browser-mode never.

Command map

kreuzcrawl scrape <url>     # single page → JSON or Markdown
kreuzcrawl crawl <url...>   # follow links, BFS, depth-bounded
kreuzcrawl map <url>        # enumerate URLs via sitemaps + link extraction
kreuzcrawl interact <url>   # browser actions: click, type, scroll
kreuzcrawl mcp              # MCP server (stdio) — auto-registered (`mcp` feature)
kreuzcrawl serve            # REST API server (`api` feature)

Batch behaviour is built into crawl: pass multiple seed URLs and the engine fans out via batch_crawl internally.

Shared flags

Flag Default Notes
--format json json or markdown.
--timeout 30000 Request timeout in milliseconds.
--browser-mode auto auto, always, or never.
--browser-endpoint — Optional CDP ws:// or wss:// URL.
--respect-robots-txt off Pass to obey robots.txt.
--config <json> — Inline JSON or @file.json to override defaults.

The --config flag accepts the full CrawlConfig schema. Anything you set explicitly on the CLI overrides the corresponding JSON field.

Scrape a single page

kreuzcrawl scrape https://example.com --format markdown

JSON output (default) carries the rendered Markdown, page metadata (PageMetadata), links by category, images, feeds, JSON-LD blocks, and HTTP response metadata. Use Markdown output when piping into a file the user will read.

See the scraping-html-to-markdown skill for the full flag surface.

Crawl a site

kreuzcrawl crawl https://example.com \
  --depth 3 --max-pages 200 --concurrent 8 --rate-limit 250 \
  --stay-on-domain --respect-robots-txt --format markdown

Crawling is BFS by default, bounded by --depth, --max-pages, and --concurrent. Per-domain politeness is enforced by --rate-limit (milliseconds between requests to the same origin).

See the crawling-a-site skill for the recommended defaults and the full flag surface.

Map URLs

kreuzcrawl map https://example.com --limit 500 --search docs --format markdown

map reads sitemap.xml (and nested sitemaps), then falls back to link extraction from the seed page. It does not render pages — use it to plan a crawl or to feed URLs into another tool.

Browser interaction

kreuzcrawl interact https://example.com \
  --actions '[{"type":"click","selector":"#load-more"},
              {"type":"wait","milliseconds":500},
              {"type":"scrape"}]'

Action types are click, type, press, scroll, wait, screenshot, executeJs, and scrape (to wait for an element, use wait with a selector field). The result wraps the final HTML under interaction.final_html. See the automating-the-browser skill for the full action schema and limits.

MCP server

When this plugin is installed in a Claude Code / Codex / Cursor / Gemini / opencode harness, the MCP server is auto-registered:

kreuzcrawl mcp

mcp is a stdio-transport server and takes no arguments. It requires a binary built with the mcp feature (see Installation).

Prefer MCP tools over shelling out when both are available:

  • Typed schemas surface argument errors before the call.
  • Results stream back as structured tool output instead of stdout text.
  • No --format juggling — the harness pulls whatever shape it needs.

Fall back to the CLI when you need to script a pipeline, capture stderr, or chain with shell tools.

Headless fallback

In --browser-mode auto (default), the engine:

  1. Fetches statically via reqwest.
  2. Detects WAF blocks (8 vendors) and JS-only shells.
  3. Re-fetches through headless Chrome with a real fingerprint when needed.

Force the browser path with --browser-mode always when you already know the page needs JS. Use --browser-mode never for hot loops where the cost of a stray Chrome launch is unacceptable.

Point --browser-endpoint ws://host:9222/devtools/browser/<id> at an already-running Chrome to skip the local launch.

See the headless-fallback skill for symptoms, costs, and external-CDP patterns.

Output formats

Mode Use when
json Downstream consumer needs metadata, links, images, etc.
markdown Human reader or LLM-context payload.

Markdown output skips metadata. If you need both, run with --format json and read result.markdown.content.

Robots, rate limits, ethics

  • --respect-robots-txt is off by default; pass it for any crawl on a host you do not own.
  • The default --rate-limit 200 already produces a polite cadence; raise it for shared hosts.
  • Identify the crawler honestly via --user-agent. Do not impersonate a browser unless the operator has approved it.

Cross-references

  • skills/crawling-a-site/SKILL.md — multi-page crawl with depth, page caps, concurrency, rate limits, and domain scoping.
  • skills/scraping-html-to-markdown/SKILL.md — single-page rendering, the Markdown output shape, and common pitfalls.
  • skills/mapping-urls/SKILL.md — map: sitemap + link URL discovery, filtering, and seeding a crawl.
  • skills/automating-the-browser/SKILL.md — interact: the full scripted action schema, limits, and result shape.
  • skills/serving-the-api/SKILL.md — serve: the Firecrawl-v1-compatible REST API server and its endpoints.
  • skills/headless-fallback/SKILL.md — when and how to force the browser backend.

04 同仓库更多 Skills

hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-review-with-annotations

Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-find-and-synthesize-context

Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-update-docs

Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-records

Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-interactive-html

Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-collaborate-threads

Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.

★ 1.6k